User updates, user deletions, list uploads, custom event, and journey processing delays
Resolved
Our engineering team has confirmed processing across all clusters is healthy and our systems have caught up on their processing backlogs. Total impact was from approximately 1:45pm PT to approximately 3:20pm PT
Posted Aug 22, 2023 - 15:35 PDT
Update
Our engineering team has rolled out a fix. Processing across all clusters is returning to a healthy rate. The team will continue to monitor. Our engineering team estimates that our systems will be caught up on their processing backlogs in approximately 30 minutes. Next update at 3:30pm PT.
Posted Aug 22, 2023 - 15:02 PDT
Identified
Our engineering team has identified an issue that is causing delays on processing user updates, user deletions, list uploads, custom event, and user journeys. Impact started at approximately 1:45pm PT. No data is being dropped only delayed. This may also delay message sends associated with impacted journeys and API calls. The team has identified a fix and is actively working on rolling it. Next update at 3pm PT.
Posted Aug 22, 2023 - 14:39 PDT
This incident affected: Cluster 5 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 6 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 8 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 9 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 10 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 11 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 12 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 13 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 14 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 15 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 16 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 17 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 18 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 19 (Emails Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 20 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 21 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), Cluster 22 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions), and Cluster 23 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions).