C22 ingestion delays

Incident Report for Iterable

Resolved

The system has finished working on backlog of data and the data is caught up. All systems are operational and have no issues.
Engineering will continue to monitor the cluster performance.
Posted May 20, 2025 - 12:02 PDT

Monitoring

Our engineering team has rolled out a fix to remediate the delays seen on c22 for user updates, list uploads, user deletion, and event processing. The system is currently working through the backlog of data and we should see it drained in a couple of hours. Engineering will continue to monitor the cluster performance until the backlog is completely drained. Next update at 11:30 am PT or sooner.
Posted May 20, 2025 - 09:34 PDT

Update

We are continuing to investigate this issue.
Posted May 20, 2025 - 08:20 PDT

Investigating

Starting at around 9 pm PT 5/19, cluster 22 began experiencing delays delays on user updates, list uploads, user deletion, and event processing. Customers could experience delays of up to 6 hours currently. The engineering team is currently investigating and working to remediate these delays. Next update at 9:30 am PT or sooner.
Posted May 20, 2025 - 07:52 PDT
This incident affected: Cluster 22 (Workflow Processing, User Updates, List Uploads, User Deletions).