Update - We have observed that the processing time returned to an acceptable rate and will continue to monitor over the weekend. Next status update Monday.
Jan 28, 2023 - 10:07 PST
Jan 28, 2023 - 10:07 PST
Monitoring - The fixes put in place to increase capacity have continued to improve database performance. We are moving this incident to the monitoring phase to ensure that improved performance continues.
Jan 27, 2023 - 22:12 PST
Jan 27, 2023 - 22:12 PST
Update - The engineering team is continuing to make improvements to the cluster, and observed a decrease in the number of query counts, resulting in the database performance continuing to improve. We are in the process of increasing capacity to further improve processing times. Next update by 10:30pm PDT.
Jan 27, 2023 - 20:40 PST
Jan 27, 2023 - 20:40 PST
Update - The database performance is improving. The engineering team will continue to monitor performance to make sure the improvement is consistent. Next update by 8:30pm PDT.
Jan 27, 2023 - 17:10 PST
Jan 27, 2023 - 17:10 PST
Update - The engineering team is continuing to work to improve the database performance. They are actively working to make the changes necessary to restore the affected services. Next update by 5pm PDT or sooner.
Jan 27, 2023 - 15:47 PST
Jan 27, 2023 - 15:47 PST
Identified - Our engineering team has identified multiple underlying causes and is continuing to implement multiple fixes to improve database performance. We will continue to monitor these changes for performance improvement. The next update will be at 3:30 PM PDT or sooner.
Jan 27, 2023 - 13:36 PST
Jan 27, 2023 - 13:36 PST
Update - We have identified a number of potential causes and our engineering team is making changes to improve the database performance. As we make these changes we will continue to monitor for the clusters performance improvement. Customers on c12 may still be experiencing the aforementioned issues while the changes are rolled out. Next update at 1:45 PM PDT or sooner.
Jan 27, 2023 - 12:38 PST
Jan 27, 2023 - 12:38 PST
Investigating - Our engineers are investigating degraded elastic search performance for our customers on c12. Customers on this cluster may experience issues such delayed journey processing for journeys utilizing filter nodes and delayed blast campaign sends. User ingestion may be slowed for processes such as list uploads and user update API calls. Certain pages of the app such as the all campaigns and list pages may also be slow to load. We will continue to investigate and will have the next update at 12:30 PM PDT or sooner.
Jan 27, 2023 - 11:41 PST
Jan 27, 2023 - 11:41 PST
Global Web Application
?
Operational
Global API
?
Operational
Global Links
?
Operational
Global System Webhooks
?
Operational
Global Partner Webhooks
?
Operational
Global Analytics Processing
?
Operational
Global Catalog
?
Operational
Cluster 5
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 6
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 8
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 9
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 10
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 11
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 12
Degraded Performance
Email Sends
Degraded Performance
Workflow Processing
Degraded Performance
Push Sends
Operational
SMS Sends
Operational
User Updates
Degraded Performance
List Uploads
Degraded Performance
User Deletions
Operational
Cluster 13
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 14
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational
List Uploads
Operational
User Deletions
Operational
Cluster 15
Operational
Email Sends
Operational
Workflow Processing
Operational
Push Sends
Operational
SMS Sends
Operational
User Updates
Operational