Our engineering team has confirmed at 2:06 PDT that all ingestion should be back to normal and is stable. This incident is now resolved.
Posted Mar 14, 2025 - 14:13 PDT
Monitoring
Engineering has finished pushing the fix. Customers on Clusters 116, 120, and 132 may still see an ingestion lag greater than 1 hour. All other Clusters have recovered. Data was never dropped. The next update is at 3 PM PDT or sooner.
Posted Mar 14, 2025 - 12:30 PDT
Identified
Engineering has put in the fix. Some clusters have fully recovered, and we are actively working on the ones that have not. For those customers that are still impacted, you will still be experiencing some delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 12:30 PM PDT or sooner.
Posted Mar 14, 2025 - 11:07 PDT
Update
The engineering team is still working on the fix, and customers on Clusters 100 and above will still be experiencing some delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 11:00 AM PDT or sooner.
Posted Mar 14, 2025 - 10:31 PDT
Investigating
Our engineering team has identified an issue causing ingestion delay for all Clusters 100 and above. We are deploying the fix right now. Customers on these impacted clusters could be experiencing delays with user updates, list uploads, user deletion, and event processing. Email sends and journey processing may see impact as well. Data is only being delayed. It is not being dropped. The engineering team is working to remediate these delays. Next update at 10:30 AM PDT or sooner.