Ingestion Delay On Subset Of Clusters
Resolved
Ingestion has returned to normal for all impacted clusters and this issue is now resolved.
Posted Nov 12, 2024 - 11:24 PST
Monitoring
The engineering team has identified the cause and remediated the delays on these subset of clusters. Most impacted ingestion topics have recovered and customers should start to see data being ingested normally again. We are still working through some backlog, so the team will continue to monitor to ensure that these delays don't recur. Next Update at 11:00 am PT or sooner?
Posted Nov 12, 2024 - 09:52 PST
Investigating
Starting at around 6:30 am PT, Iterable Engineers were notified that lag on our ingestion topics was growing on a subset of clusters. Customers on these impacted clusters could be experiencing delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 10:00 an PT or Sooner.
Posted Nov 12, 2024 - 09:18 PST
This incident affected: Cluster 14 (Email Sends, Workflow Processing, User Updates, List Uploads, User Deletions), Cluster 16 (Email Sends, Workflow Processing, User Updates, List Uploads, User Deletions), Cluster 22 (Email Sends, Workflow Processing, User Updates, List Uploads, User Deletions), Cluster 25 (Email Sends, Workflow Processing, User Updates, List Updates, User Deletions), Cluster 115 (Email Sends, Workflow Processing, User Updates, List Updates, User Deletions), and Cluster 125 (Email Sends, Workflow Processing, User Updates, List Updates, User Deletions).