The second phase of remediations has concluded for the erroneous Gmail unsubscribes. We appreciate your patience and understanding. If you have any further questions, please do not hesitate to contact your account manager or email support@iterable.com
Posted Dec 21, 2020 - 16:03 PST
Update
Our engineers are currently working towards rolling out the second phase of the remediation process against better precision for unsubscribes based on users' lifetime subscribe/unsubscribe events.
Next update at 3 pm PST.
Posted Dec 21, 2020 - 09:07 PST
Update
We have finished the first half of remediations for the erroneous Gmail unsubscribes for a majority of our customers. This has restored user unsubscribe states to the previous ones before the Gmail outage.
The second phase of remediations is when we will be validating the restored subscriptions against previous user lifetime subscription events to maximize accuracy. This work will continue throughout the weekend. Next update will be Monday morning December 21st at 9:00 AM PST.
If you have any questions please reach out to your account managers or contact support@iterable.com
Posted Dec 18, 2020 - 18:42 PST
Monitoring
As of 12:15 pm PST our engineering team has completed all testing to confirm stability and data integrity of our proposed fix. We have begun rolling said fix across our customer base to revert the erroneous unsubscribes caused by the Google Gmail outage. This work is expected to last through the remainder of the day since the time for the fix to finish varies by customer. If you have any further questions please do not hesitate to reach out to your account manager or support@iterable.com.
We will post a status page update once the fix has been completed across all customers. For customers that have proactively completed internal fixes on their own this should not impact the work that was completed.
Posted Dec 18, 2020 - 12:20 PST
Update
Engineering has completed their initial preparations for remediation. The team is now thoroughly testing their solution to ensure continued platform stability and data integrity. This is taking some time, but is critical to the success of the remediation. As a result, the plan is now to carry out all updates tomorrow, with the expectation that the process will be complete by the end of day tomorrow (Friday). Thank you for your patience.
Posted Dec 17, 2020 - 13:48 PST
Update
Our engineering team has begun remediation efforts to address the erroneous unsubscribes caused by the inaccurate bounce notifications resulting from Google's mail outages. Given the large number of recipients impacted, it is expected to take some time to fully-process. Our current estimate is that the bulk of updates should be completed tomorrow (Thursday). We'll be sure to provide updates to that estimate as and when we have them. We appreciate everyone's patience with the process as we work through this difficult situation.
Posted Dec 16, 2020 - 21:13 PST
Identified
Starting on Monday December 14th, Google experienced a series of platform outages which impacted their Gmail service, for both GSuite and regular Gmail recipients worldwide. Google erroneously generated hard bounces indicating that legitimate mailboxes did not exist. The Google issues are now resolved. More information can be found here: https://www.google.com/appsstatus#hl=en&v=status
In Iterable, a hard bounce will automatically unsubscribe the recipient from that channel, and customers have seen unusual spikes in bounce rates and unsubscribes as a result of the issue. We are actively working on remediation efforts to roll back those unsubscribes. We are also working with our MTA partners who are going through a similar effort. Due to the scale of the impact, complete resolution could take some time. Once we have more information, we will provide a further update.
Posted Dec 16, 2020 - 10:22 PST
This incident affected: Global Analytics Processing.