Degraded workflow and API performance

Incident Report for Iterable

Resolved

Scheduled campaigns creation has returned to working as expected and we are no longer seeing ingestion delay on Catalogs. If you have any additional questions please contact your account manager or email support@iterable.com
Posted Sep 27, 2021 - 09:00 PDT

Update

Our engineers have identified scheduled campaign creation between the hours of 8:53 pm PDT to 7:20 am PDT were delayed in creation. These campaigns are now in the process of creation and sending.
Posted Sep 27, 2021 - 07:41 PDT

Update

Catalog ingestion issues are now recovering, and we anticipate fully processing the ingestion backlog within an hour. The next update at 9 am PDT.
Posted Sep 27, 2021 - 07:03 PDT

Update

For the ongoing Catalog ingestion delay: Our engineers are continuing to investigate possible remedies for the ingestion delay. Next update at 7 am PDT
Posted Sep 27, 2021 - 05:56 PDT

Update

Status page: Catalog ingestion - Our Engineers are still looking to remediate. Next update will be at 06:00am PST
Posted Sep 27, 2021 - 04:35 PDT

Update

We are continuing to work on a fix for this issue.
Posted Sep 27, 2021 - 02:49 PDT

Update

Wide spread API errors have recovered. For Catalog ingestion, our Engineers are looking to remediate.
Posted Sep 27, 2021 - 02:49 PDT

Update

We believe that sends are now functioning normally. Catalog ingestion is stopped with a fix being readied.
For API Errors, we believe are related to the EBS issues however we're still investigating.
Posted Sep 27, 2021 - 01:33 PDT

Update

We are are seeing failures on API calls. Our engineers are currently investigating.
Posted Sep 27, 2021 - 00:44 PDT

Update

While we are seeing improvements in the affected components, workflow processing, data ingestion, and message sends continue to experience delays as we work through the queue backlogs. Work continues with our 3rd party vendors on remediations as our engineering team also continues to implement fixes to the affected systems. Next update will be at 1:30 AM PST
Posted Sep 27, 2021 - 00:03 PDT

Identified

We are seeing improvements in processing for email sends and workflows as our system continues to work through the backlog caused by the ingestion delays. This means customers may still be experiencing delays with workflow entrances, processing, user events, and message sends. We are continuing work with our 3rd party provider on further remediations. Next update at 12:00 AM PST
Posted Sep 26, 2021 - 22:56 PDT

Investigating

At approximately 7:35 PM PST, due to a downstream incident suffered by a third party vendor, workflow processing, email sends, and data ingestion is delayed. During this time user processing in workflows, along with email sends including scheduled campaigns are delayed but not dropped. We will continue working with our provider on mitigating the issue. Next update is at 11:00 PM PST
Posted Sep 26, 2021 - 21:32 PDT
This incident affected: Cluster 5 (Email Sends, Workflow Processing), Cluster 6 (Email Sends, Workflow Processing), Cluster 8 (Email Sends, Workflow Processing), Cluster 9 (Email Sends, Workflow Processing), Cluster 10 (Email Sends, Workflow Processing), Cluster 11 (Email Sends, Workflow Processing), Cluster 12 (Email Sends, Workflow Processing), Cluster 13 (Email Sends, Workflow Processing), Cluster 14 (Email Sends, Workflow Processing), Cluster 15 (Email Sends, Workflow Processing), Cluster 16 (Email Sends, Workflow Processing), Cluster 17 (Email Sends, Workflow Processing), Cluster 18 (Email Sends, Workflow Processing), Cluster 19 (Emails Sends, Workflow Processing), Cluster 20 (Email Sends, Workflow Processing), Cluster 21 (Email Sends, Workflow Processing), and Global API Success, Global Analytics Processing, Global Catalog.