Update - Our engineering team has continued to monitor the fix and we have seen no further issues with Journey entrances and processing. Total impact time was approximately from 5 am PDT until 8:36 am PDT. During this time no users were dropped, only delayed on their Journey entrances/ triggers. We will continue to monitor the health of Journey ingestion and processing before resolving this incident.
Dec 01, 2022 - 09:51 PST
Dec 01, 2022 - 09:51 PST
Monitoring - Our team has identified the cause for the memory increase and applied a fix. The error rates are decreasing and the system is catching up. Operation has improved for workflow processing and we are currently monitoring the memory utilization.
The next update at 9:30am PT or earlier.
Dec 01, 2022 - 09:02 PST
The next update at 9:30am PT or earlier.
Dec 01, 2022 - 09:02 PST
Investigating - As of approximately 7am PT due to workflow memory issues there has been a degradation in workflow processing across all customers. Our engineering team is investigating the root cause while allocating resources to reduce customer impact.
Next update at 9am PT or sooner.
Dec 01, 2022 - 08:24 PST
Next update at 9am PT or sooner.
Dec 01, 2022 - 08:24 PST
Global Web Application
?
Operational
Global API
?
Operational
Global Links
?
Operational
Global System Webhooks
?
Operational
Global Partner Webhooks
?
Operational
Global Analytics Processing
?
Operational
Global Catalog
?
Operational
Cluster 5
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 6
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 8
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 9
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 10
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 11
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 12
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 13
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 14
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 15
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational