After monitoring the clusters, we have reached a resolution on this incident. If you experience any unusual behavior with the Iterable platform, please open a support ticket by sending an email to support@iterable.com.
Posted Aug 22, 2018 - 22:14 PDT
Monitoring
We've completed remediation tasks on all affected infrastructure. APIs, outbound messages and workflows all appear to functioning correctly for all customers - however one server cluster is draining slowly, so we're still focusing efforts on that cluster. It may take some time for all queues to completely drain, which means that workflow tasks may still be delayed for the next couple of hours. Please let us know if you see anything unusual. Thank you for your patience today. We will continue to monitor for a few more hours, and will move this to resolved once we see all queues completely drained.
Posted Aug 22, 2018 - 16:17 PDT
Update
We’ve completed a cluster restart which has improved the throughput of one cluster (C5), but other clusters are showing signs of slowdown and may need to be restarted to resolve. We are working to further isolate the clusters from each other so we can minimize impact of any further changes we need to make. The next update will be at 4 PM Pacific Time or sooner.
Posted Aug 22, 2018 - 14:54 PDT
Update
Our cluster restart is continuing - ETA for completion is by 1:15pm. Next update will either be once we've recovered all services, or by 2:30pm.
Posted Aug 22, 2018 - 12:47 PDT
Update
We're continuing to work on bringing the C5 queue server cluster back to health. It is currently restarting, which takes 30-45 minutes. All other clusters are consuming events properly. Next update: 1pm Pacific.
Posted Aug 22, 2018 - 12:03 PDT
Identified
We've identified the root cause of the issues, and are now working to mitigate the impact. This has begun by allowing the non-affected queues to start draining. Following that, we will be able to start draining the C5 queue. Next update by 11:45am Pacific Time.
Posted Aug 22, 2018 - 11:05 PDT
Update
Our engineers are seeing significant issues with one specific server cluster that processes event queues. This cluster (C5) is causing other clusters to back up, so we've pulled C5 out of the pool. This should allow all the other queues to drain, while we work on C5's specific issues. Please reach out to your CSM only if you're experiencing issues, as this is not affecting all customers. CSMs are also proactively reaching out to customers who are affected at this time. Next update by 11am Pacific Time.
Posted Aug 22, 2018 - 10:33 PDT
Update
We are continuing to investigate this issue.
Posted Aug 22, 2018 - 09:46 PDT
Investigating
Some of our customers are reporting slow email sending, users not progressing through workflows and issues with API calls. Our engineering team have confirmed this though analysis of error logs. We don't think this affects everyone, and seems to be isolated to one specific server cluster, C5. However we are still in the early stages of the investigation, so impact may be more widespread. Next update by 10:15am Pacific Time.