Workflows are now processing normally. This incident has now concluded. Any additional issues should be reported to support@iterable.com, and we will follow up as necessary. Thank you for your patience today, and we appreciate all our customers for partnering with Iterable.
Posted Jul 25, 2018 - 21:37 PDT
Update
Workflows are back to operational and are no longer degraded. The backlog is still being processed and we will update this page when the backlog has been completely processed
Posted Jul 25, 2018 - 19:57 PDT
Update
The affected queues are continuing to recover and are processing workflows. The backlog of events is being reduced and we will continue to monitor and provide updates.
Posted Jul 25, 2018 - 16:49 PDT
Monitoring
The server cluster that was impacted and was causing stuck workflows for some customers is now processing workflows again. There remains a backlog of events to process, but the system is working its way through the backlog, and we are expecting things to return to normal again as the afternoon progresses. Next update: 4:15pm Pacific.
Posted Jul 25, 2018 - 15:12 PDT
Identified
We have identified the factors which have caused users to not progress through workflows, and are now working with one of our technology partners to fix the issue. We expect to see those workflows start to function again within the next 15-30 minutes, although there will still be delays through the afternoon as the queues recover. We'll provide another update within the next 30 minutes.
Posted Jul 25, 2018 - 14:37 PDT
Update
We apologize for the delay in providing this update - work is progressing and we wanted to provide our customers with some concrete information, which required waiting a little longer than planned to send an update.
The team is still digging into the cause of the workflow issues, and are investigating a highly unusual spike in events being passed to our queuing infrastructure. They have also added additional compute resources to work through the queues at a higher rate. We are still aiming for a resolution by 2pm, but that may change. Should that not be the case, the next update will be on or before 2:30pm.
Posted Jul 25, 2018 - 13:41 PDT
Update
The engineering team have narrowed down the root cause of the current workflow issues to a few culprits, and are actively investigating each right now. Although it might change, we expect to be at or near resolution within an hour. Next update: 1:30pm Pacific.
Posted Jul 25, 2018 - 13:02 PDT
Update
Our engineers are continuing to troubleshoot the issues we're seeing with workflows for some customers. It is worth noting that this is only impacting a subset of customers, those that are handled by a specific server cluster. We can't provide an ETA for resolution just yet, but we will provide another update by 1pm Pacific.
Posted Jul 25, 2018 - 12:37 PDT
Investigating
We have received reports from some customers that users are not being sent through workflows correctly. Our engineering team is actively investigating this issue, and it appears to be limited to one specific server cluster, meaning that only those customers assigned to that cluster could be impacted. We're working with our technology partners to fix the issue, and we expect to have an ETA for a fix by the next update, which will be on or before 12:30pm Pacific Time.