AWS us-east1 outage causing Email sending, API processing, and Analytics delays for a subset of customers
Resolved
Message send rates are back to normal and we are no longer seeing delays in sends. If you have any additional questions please contact your account manager or email support@iterable.com.
Posted Dec 07, 2021 - 22:37 PST
Update
We are continuing to monitor and see improvements in all impacted areas due to the ongoing AWS issues. Next update 11 PM PST or sooner
Posted Dec 07, 2021 - 20:09 PST
Update
Message sends are returning to normal levels, we are continuing to work through the backlog of delayed messages due to the ongoing AWS issues. Next update by 8 PM PST or sooner
Posted Dec 07, 2021 - 18:00 PST
Identified
Message sends have begun processing slowly as we begin working through the backlog of delayed messages due to the ongoing AWS issues. Our engineering team is continuing to investigate and work with our 3rd party downstream providers on mitigating the send delays. Next update by 6 PM PST
Posted Dec 07, 2021 - 16:51 PST
Investigating
As a result of the the downstream effects of the AWS us-east1 outage, we are now investigating reports of platform wide send failures across all channels. Our engineering team is investigating and we will have an update shortly.
Posted Dec 07, 2021 - 16:01 PST
Monitoring
A fix has been deployed per AWS to resolve the AWS us-east1 outage. We are continuing to monitor and see improvements in all impacted areas. Some customers may still experience delays while we work through the backlog of sends and data processing. We will continue to monitor the situation and work with our downstream providers on any additional remediations that may be needed. Next update at 4:30 PM PST or sooner.
Posted Dec 07, 2021 - 14:20 PST
Update
We are continuing to work on mitigating the issues caused by the AWS us-east1 outage. While we are seeing some improvements in message send processing, workflow processing, and analytics, some customers may continue to experience delays. API ingestion looks to be recovered. We will continue working with our downstream providers on additional mitigations. Next update at 2:00 PM PST.
Posted Dec 07, 2021 - 12:35 PST
Identified
We are continuing to work on mitigating the issues caused by the AWS us-east1 outage. Some customers may still be experiencing delays on email sends, API processing, workflow processing, and analytics. Data is only delayed, not lost. Next update at 12:30 PM PST
Posted Dec 07, 2021 - 11:32 PST
Investigating
We are investigating reports that due to a downstream outage of a 3rd party provider, a subset of customers may be experiencing platform degradation in the form of delayed email sends, delayed API calls, and delayed analytics processing. We are continuing to investigate the full impact and our engineering team is working on mitigations. Next update at 11:30 AM PST
Posted Dec 07, 2021 - 10:41 PST
This incident affected: Global Web Application, Global API, Global Analytics Processing, Cluster 5 (Email Sends, Workflow Processing, Push Sends), Cluster 6 (Email Sends, Workflow Processing, Push Sends), Cluster 8 (Email Sends, Workflow Processing, Push Sends), Cluster 9 (Email Sends, Workflow Processing, Push Sends), Cluster 10 (Email Sends, Workflow Processing, Push Sends), Cluster 11 (Email Sends, Workflow Processing, Push Sends), Cluster 12 (Email Sends, Workflow Processing, Push Sends), Cluster 13 (Email Sends, Workflow Processing, Push Sends), Cluster 14 (Email Sends, Workflow Processing, Push Sends), Cluster 15 (Email Sends, Workflow Processing, Push Sends), Cluster 16 (Email Sends, Workflow Processing, Push Sends), Cluster 17 (Email Sends, Workflow Processing, Push Sends), Cluster 18 (Email Sends, Workflow Processing, Push Sends), Cluster 19 (Emails Sends, Workflow Processing, Push Sends), Cluster 20 (Email Sends, Workflow Processing, Push Sends), and Cluster 21 (Email Sends, Workflow Processing, Push Sends).