Update - We have observed that the processing time returned to an acceptable rate and will continue to monitor over the weekend. Next status update Monday.
Jan 28, 2023 - 10:07 PST
Monitoring - The fixes put in place to increase capacity have continued to improve database performance. We are moving this incident to the monitoring phase to ensure that improved performance continues.
Jan 27, 2023 - 22:12 PST
Update - The engineering team is continuing to make improvements to the cluster, and observed a decrease in the number of query counts, resulting in the database performance continuing to improve. We are in the process of increasing capacity to further improve processing times. Next update by 10:30pm PDT.
Jan 27, 2023 - 20:40 PST
Update - The database performance is improving. The engineering team will continue to monitor performance to make sure the improvement is consistent. Next update by 8:30pm PDT.
Jan 27, 2023 - 17:10 PST
Update - The engineering team is continuing to work to improve the database performance. They are actively working to make the changes necessary to restore the affected services. Next update by 5pm PDT or sooner.
Jan 27, 2023 - 15:47 PST
Identified - Our engineering team has identified multiple underlying causes and is continuing to implement multiple fixes to improve database performance. We will continue to monitor these changes for performance improvement. The next update will be at 3:30 PM PDT or sooner.
Jan 27, 2023 - 13:36 PST
Update - We have identified a number of potential causes and our engineering team is making changes to improve the database performance. As we make these changes we will continue to monitor for the clusters performance improvement. Customers on c12 may still be experiencing the aforementioned issues while the changes are rolled out. Next update at 1:45 PM PDT or sooner.
Jan 27, 2023 - 12:38 PST
Investigating - Our engineers are investigating degraded elastic search performance for our customers on c12. Customers on this cluster may experience issues such delayed journey processing for journeys utilizing filter nodes and delayed blast campaign sends. User ingestion may be slowed for processes such as list uploads and user update API calls. Certain pages of the app such as the all campaigns and list pages may also be slow to load. We will continue to investigate and will have the next update at 12:30 PM PDT or sooner.
Jan 27, 2023 - 11:41 PST
Global Web Application ? Operational
Global API ? Operational
Global Links ? Operational
Global System Webhooks ? Operational
Global Partner Webhooks ? Operational
Global Analytics Processing ? Operational
Global Catalog ? Operational
Cluster 5 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational
90 days ago
100.0 % uptime
Today
List Uploads Operational
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 6 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational
90 days ago
100.0 % uptime
Today
List Uploads Operational
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 8 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational
90 days ago
100.0 % uptime
Today
List Uploads Operational
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 9 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational
90 days ago
100.0 % uptime
Today
List Uploads Operational
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 10 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational
90 days ago
100.0 % uptime
Today
List Uploads Operational
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 11 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational
90 days ago
100.0 % uptime
Today
List Uploads Operational
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 12 Degraded Performance
90 days ago
100.0 % uptime
Today
Email Sends Degraded Performance
90 days ago
100.0 % uptime
Today
Workflow Processing Degraded Performance
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Degraded Performance
90 days ago
100.0 % uptime
Today
List Uploads Degraded Performance
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 13 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational
90 days ago
100.0 % uptime
Today
List Uploads Operational
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 14 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational
90 days ago
100.0 % uptime
Today
List Uploads Operational
90 days ago
100.0 % uptime
Today
User Deletions Operational
90 days ago
100.0 % uptime
Today
Cluster 15 Operational
90 days ago
100.0 % uptime
Today
Email Sends Operational
90 days ago
100.0 % uptime
Today
Workflow Processing Operational
90 days ago
100.0 % uptime
Today
Push Sends Operational
90 days ago
100.0 % uptime
Today
SMS Sends Operational
90 days ago
100.0 % uptime
Today
User Updates Operational