Bulk ingestion lag has subsided. There are no longer any delays regarding list uploads or user updates for the subset of customers impacted today.
Posted Oct 21, 2019 - 16:57 PDT
Update
Bulk endpoints are continuing to see some lag but are steadily improving. Next update by 5pm Pacific Time.
Posted Oct 21, 2019 - 15:00 PDT
Monitoring
Most of the non-bulk ingestion lag is now gone. Bulk endpoints will still see some lag but it continues to improve. Next update by 3pm Pacific Time.
Posted Oct 21, 2019 - 13:10 PDT
Update
The impacted server cluster is now making progress on catching up on ingestion lag. The current expected time for catching up with non-bulk ingestion is about two hours. There may still be some delays in workflow processing and campaign sends. Next update by 12pm Pacific Time.
Posted Oct 21, 2019 - 11:53 PDT
Update
The team is continuing to work on getting the affected database cluster back to a healthy state. It appears as though the root cause was a faulty node within the database cluster that caused knock-on performance issues. We will be able to provide more information in a post-mortem document when it becomes available.
Impacted customers might be seeing some issues with functionality in the web application, such as errors or slowness with creating segmentation. Data ingestion (including some APIs), workflow processing and sending are all still degraded for some customers. Next update by 11:45am Pacific Time.
Posted Oct 21, 2019 - 10:41 PDT
Update
The affected database is still recovering. The delays in data ingestion, workflow processing, and campaign sends are still present. The next update will be at 10:30 AM Pacific time
Posted Oct 21, 2019 - 09:33 PDT
Update
We are continuing to work on a fix for this issue.
Posted Oct 21, 2019 - 09:15 PDT
Update
The impacted database cluster is continuing to recover. Some customers may still be seeing delays in data ingestion, workflow processing and campaign sends. We're expecting both the delays and number of impacted customers to continue to decrease. Next update by 9:30am Pacific Time.
Posted Oct 21, 2019 - 08:27 PDT
Identified
One of our database clusters experienced an issue around 6:40am Pacific Time which impacted a subset of customers, those that were on that particular cluster. We saw data ingestion delays, along with slow processing of workflows and campaign sends. We have identified the issue and the cluster is now recovering, but will take some time before it returns to normal. Next update by 8:30am Pacific Time.
Posted Oct 21, 2019 - 07:40 PDT
This incident affected: Global Web Application and Global API Success.