The engineering team has confirmed that journey processing is healthy and our systems have caught up on their processing backlogs. For the most part, impact was between approximately 3:00am PT and 9:20am PT today. Clusters 6 and 9 took a little longer to recover, so impact for those was between approximately 3:00am PT and 12:00pm PT.
Posted Aug 27, 2023 - 13:10 PDT
Monitoring
The engineering team rolled out a remediation that has brought journey processing back to a normal level. The team will continue to monitor the situation. Impact was between approximately 3:00am PT and 9:20am PT today.
Posted Aug 27, 2023 - 10:03 PDT
Update
We are continuing to investigate this issue.
Posted Aug 27, 2023 - 09:11 PDT
Investigating
The engineering team has identified an issue in our journey processing engine with implications across our customer base. The specific impact is delayed processing of journeys across multiple clusters. Impact began at approximately 3am PT today. No data has been lost. Next update at 10:00am PT.
Posted Aug 27, 2023 - 09:10 PDT
This incident affected: Global Web Application and Global API.