Ingestion services degraded - impacting campaigns, workflows, APIs

Incident Report for Iterable

Resolved

After monitoring our ingestion queues, it appears as though we've reached resolution on this incident. Should you experience any unusual behavior with the Iterable platform, please open a support ticket by sending an email to support@iterable.com. We appreciate your patience this afternoon and evening.
Posted Aug 14, 2018 - 20:36 PDT

Update

We are continuing to monitor for any further issues.
Posted Aug 14, 2018 - 19:13 PDT

Monitoring

We have deployed a fix that should resolve this issue. We will continue to monitor and send an update when we are confident this issue has been resolved.
Posted Aug 14, 2018 - 19:07 PDT

Identified

We have identified the issue which has been caused by malformed messages entering the the ingestion queues. We are in the process of removing those messages. This will allow messages that were placed on the queue after this affected period to be processed. We will continue to monitor this issue and update this page regularly.
Posted Aug 14, 2018 - 17:50 PDT

Investigating

Our engineering team is currently troubleshooting an issue with our ingestion servers. These servers touch a number of areas of the Iterable platform, and customers are seeing impact in various ways: API calls may fail, workflows may not progress, analytics might not be correct and messages may not send. The impact is significant, and it is being actively worked on by a number of engineers. Next update by 6pm Pacific Time.
Posted Aug 14, 2018 - 17:28 PDT
This incident affected: Global API Success, Global System Webhooks, and Global Analytics Processing.