Degraded workflow and API performance
Resolved
Scheduled campaigns creation has returned to working as expected and we are no longer seeing ingestion delay on Catalogs. If you have any additional questions please contact your account manager or email support@iterable.com
Posted Sep 27, 2021 - 09:00 PDT
Update
Our engineers have identified scheduled campaign creation between the hours of 8:53 pm PDT to 7:20 am PDT were delayed in creation. These campaigns are now in the process of creation and sending.
Posted Sep 27, 2021 - 07:41 PDT
Update
Catalog ingestion issues are now recovering, and we anticipate fully processing the ingestion backlog within an hour. The next update at 9 am PDT.
Posted Sep 27, 2021 - 07:03 PDT
Update
For the ongoing Catalog ingestion delay: Our engineers are continuing to investigate possible remedies for the ingestion delay. Next update at 7 am PDT
Posted Sep 27, 2021 - 05:56 PDT
Update
Status page: Catalog ingestion - Our Engineers are still looking to remediate. Next update will be at 06:00am PST
Posted Sep 27, 2021 - 04:35 PDT
Update
We are continuing to work on a fix for this issue.
Posted Sep 27, 2021 - 02:49 PDT
Update
Wide spread API errors have recovered. For Catalog ingestion, our Engineers are looking to remediate.
Posted Sep 27, 2021 - 02:49 PDT
Update
We believe that sends are now functioning normally. Catalog ingestion is stopped with a fix being readied.
For API Errors, we believe are related to the EBS issues however we're still investigating.
Posted Sep 27, 2021 - 01:33 PDT
Update
We are are seeing failures on API calls. Our engineers are currently investigating.
Posted Sep 27, 2021 - 00:44 PDT
Update
While we are seeing improvements in the affected components, workflow processing, data ingestion, and message sends continue to experience delays as we work through the queue backlogs. Work continues with our 3rd party vendors on remediations as our engineering team also continues to implement fixes to the affected systems. Next update will be at 1:30 AM PST
Posted Sep 27, 2021 - 00:03 PDT
Identified
We are seeing improvements in processing for email sends and workflows as our system continues to work through the backlog caused by the ingestion delays. This means customers may still be experiencing delays with workflow entrances, processing, user events, and message sends. We are continuing work with our 3rd party provider on further remediations. Next update at 12:00 AM PST
Posted Sep 26, 2021 - 22:56 PDT
Investigating
At approximately 7:35 PM PST, due to a downstream incident suffered by a third party vendor, workflow processing, email sends, and data ingestion is delayed. During this time user processing in workflows, along with email sends including scheduled campaigns are delayed but not dropped. We will continue working with our provider on mitigating the issue. Next update is at 11:00 PM PST
Posted Sep 26, 2021 - 21:32 PDT
This incident affected: Cluster 6 (Email Sends, Workflow Processing), Cluster 14 (Email Sends, Workflow Processing), Cluster 18 (Email Sends, Workflow Processing), Cluster 16 (Email Sends, Workflow Processing), Cluster 9 (Email Sends, Workflow Processing), Cluster 5 (Email Sends, Workflow Processing), Cluster 10 (Email Sends, Workflow Processing), Cluster 19 (Emails Sends, Workflow Processing), Cluster 12 (Email Sends, Workflow Processing), Cluster 11 (Email Sends, Workflow Processing), Cluster 15 (Email Sends, Workflow Processing), Cluster 17 (Email Sends, Workflow Processing), Cluster 13 (Email Sends, Workflow Processing), Cluster 8 (Email Sends, Workflow Processing), Cluster 21 (Email Sends, Workflow Processing), Cluster 20 (Email Sends, Workflow Processing), and Global API, Global Analytics Processing, Global Catalog.