We've completed some key upgrades aimed at addressing the root causes of today's (and other recent) issues, so this incident is now resolved. Again, thank you for your patience.
Posted Aug 31, 2018 - 18:06 PDT
Update
We're going to continue holding this incident in the "monitoring" stage for now. This is because our engineering team is deploying some very significant upgrades to address the root causes of issues that customers have experienced both today and previously. Once those deployments are complete, and we're satisfied with the upgrade, we'll mark this as resolved. Most likely that will be tomorrow (Saturday). Thank you for your patience - we're optimistic that the major work done today will make a meaningful improvement to overall platform stability.
Posted Aug 31, 2018 - 16:50 PDT
Update
We are not seeing any new issues, and queues appear to have returned to normal. As long as we don't see any further delays, the next update will be to mark this as resolved, by 5pm Pacific Time.
Posted Aug 31, 2018 - 14:37 PDT
Monitoring
“The API errors issues are now showing as resolved, and the Engineering team is continuing to work on the affected queues. If you or your team continues to experience issues with our API please submit a ticket to support@iterable.com or submit a chat through our UI. Next update to be at 2:30 PM PST.
Posted Aug 31, 2018 - 13:13 PDT
Update
We've completely drained some of the affected queues but are finding some campaigns still in a stuck state. We are working on an approach to resolve those campaigns. In addition, we have found an increasing rate of API errors for a small number of customers, which we are now working on resolving. Next update by 12:30pm Pacific Time.
Posted Aug 31, 2018 - 11:08 PDT
Update
We've continued to move load away from impacted queue clusters, and queue backlogs are continuing to improve. There are still some significant backlogs but we're seeing a steady decline. Our engineering team is also working on some infrastructure and code changes today to ensure we don't experience these kind of backlogs in the future. Next update will be before 11am Pacific Time.
Posted Aug 31, 2018 - 09:58 PDT
Identified
We have identified the issue that is causing campaigns to send slowly. Right now we're working to redistribute the load on the affected queue clusters, and this is allowing queues to begin to make meaningful progress. We expect to see the situation continue to improve, and will update again prior to 10am Pacific Time.
Posted Aug 31, 2018 - 08:56 PDT
Investigating
We've received reports from some customers that campaigns are deploying very slowly. So far it appears to be limited to a couple of queue servers, which means only some customers are impacted. We're investigating this right now, and will provide an update by 9am Pacific Time.