Slow workflows
Incident Report for Iterable
Resolved
We've been monitoring workflows all morning and they are continuing to progress as expected. At this point, the incident is resolved. As always, if you see any ongoing issues please reach out to us. Premier Support customers can leverage their assigned support engineer, Plus Support customers can open a live chat, and Standard Support customers can either open a ticket in the web app, or email us at support@iterable.com. Thank you for your patience through this incident.
Posted Oct 29, 2020 - 12:58 PDT
Monitoring
The underlying workflow processing issues have been addressed and workflows backlogs are recovering. Our engineering team is monitoring the systems for any adverse effects. Unless there are any changes, we plan on closing out this incident tomorrow.
Posted Oct 28, 2020 - 23:58 PDT
Update
Workflow processing has improved and our engineering team is monitoring the systems. We will provide another update at 12AM PT.
Posted Oct 28, 2020 - 21:58 PDT
Update
Our engineers have identified some possible causes of the slow workflows and have taken some remediation steps. We're starting to see an improvement in the speed of workflow processing. The team is continuing to work on the underlying issues, and we will provide another update by 10pm PT.
Posted Oct 28, 2020 - 20:09 PDT
Update
Our engineers are still working to remediate the issues from multiple angles. We'll continue to provide updates. Next update by 8pm PT.
Posted Oct 28, 2020 - 17:56 PDT
Update
No major updates right now on the issues we're seeing with slow workflows. Our engineers are still hard at work looking at possible root causes, and will continue to do so. We'll provide updates throughout the evening, with the next coming by 6pm PT.
Posted Oct 28, 2020 - 16:51 PDT
Update
We're still working on identifying the root cause of the workflow issues. It does appear as though the problem is not limited to trigger nodes, and could be seen in other workflow nodes too. Next update by 4:45pm PT.
Posted Oct 28, 2020 - 15:44 PDT
Update
Our engineers are continuing to work on the issue. We have determined that the issue is limited to triggered workflows. This means that other workflows should be unaffected. We'll post another update by 3:40pm PT.
Posted Oct 28, 2020 - 14:41 PDT
Investigating
We're currently seeing increased delays in processing workflows. Our engineering team is actively working to identify a root cause. We'll provide another update no later than 2:30pm PT.
Posted Oct 28, 2020 - 13:41 PDT