Email send delays
Incident Report for Iterable
Resolved
We have continued to monitor the message queues and have seen no additional delays since our fix was deployed and the backlog was processed. We will continue to watch closely so that no further issues occur. If you have any questions please do not hesitate to reach out to your account manager or email support@iterable.com. An RCA will be available upon request.
Posted May 11, 2022 - 16:08 PDT
Monitoring
Engineering has completed deploying a fix to mitigate the send delays and we are seeing an improvement as we process through the message backlog. We will continue to monitor our system to make sure there are no further issues and delays. Next update will be at 4 PM PDT or sooner.
Posted May 11, 2022 - 13:56 PDT
Update
We have noticed our send queues in a healthier state since our last adjustment. The engineering team has identified the fix to this issue but is working on getting that finished and deployed. The next update will be at 2 PM PDT if not sooner.
Posted May 11, 2022 - 10:28 PDT
Update
The engineering team has made some changes that appear to be alleviating the issue. We are continuing to monitor this issue as the backlog drains. The next update will be 10 AM PDT if not sooner.
Posted May 11, 2022 - 07:16 PDT
Update
Our Engineering team is still working on this. No update at this time. Next update at 7 AM PDT or sooner.
Posted May 11, 2022 - 06:14 PDT
Update
Our Engineering team is continuing to work on mitigations to improve send rates for impacted customers. Next update at 6 AM PDT or sooner.
Posted May 11, 2022 - 05:02 PDT
Identified
Although Engineering provided a fix, we are seeing a build up occur in the backend resulting in delays on campaign sends for customers utilizing snippets and catalogs. Our engineering team are continuing to work on mitigations to improve send rates for impacted customers. Next update will be at 5 AM PDT or sooner.
Posted May 11, 2022 - 04:03 PDT
Monitoring
The Engineering team has put mitigations into place, and we are no longer experiencing email send delays. We will continue to monitor the issue to make sure the mitigations are successful.
Posted May 10, 2022 - 19:35 PDT
Update
Our engineering team has put further mitigations in place and we are seeing the send delays decrease for customers as we continue to work through the backlog. Next update will be at 8 PM PDT or sooner.
Posted May 10, 2022 - 18:03 PDT
Update
We are still seeing delays on campaign sends for customers utilizing snippets and catalogs. Our engineering team is continuing to work on mitigations to improve send rates for impacted customers. Next update at 6 PM PDT or sooner.
Posted May 10, 2022 - 16:31 PDT
Identified
We are currently seeing a delay in email sends, specifically in campaigns that use Snippets or Catalogs. The engineering team is mitigating the issue, and will monitor the results. Next update at 4:30pm PT
Posted May 10, 2022 - 15:30 PDT
This incident affected: Cluster 5 (Email Sends), Cluster 6 (Email Sends), Cluster 8 (Email Sends), Cluster 9 (Email Sends), Cluster 10 (Email Sends), Cluster 11 (Email Sends), Cluster 12 (Email Sends), Cluster 13 (Email Sends), Cluster 14 (Email Sends), Cluster 15 (Email Sends), Cluster 16 (Email Sends), Cluster 17 (Email Sends), Cluster 18 (Email Sends), Cluster 19 (Emails Sends), Cluster 20 (Email Sends), and Cluster 21 (Email Sends).