Investigating - A subset of our customers is currently experiencing delays with data ingestion of up to 2 hours. It is impacting campaign sends, event tracking, user updates, and CSV uploads. Our engineering team is working diligently to resolve this issue. Next update by 1pm PDT.
Jun 26, 11:19 PDT
Workflow processing is now completely caught up, resolving this incident. If you have any questions or concerns, please don't hesitate to reach out to us at email@example.com
Jun 25, 19:48 PDT
Data ingestion is now completely caught up, but it has caused a knock-on effect of backing up workflow processing, which may be delayed up to two hours in some cases. We're monitoring to ensure that this delay is continuing to decrease. Next update by 8pm Pacific Time.
Jun 25, 17:36 PDT
We have identified and corrected the cause the of the data ingestion lag, and the lag has now almost completely caught up. Next update by 5:30pm Pacific Time.
Jun 25, 16:31 PDT
We're seeing noticeable lag on data ingestion, primarily with list uploads and event ingestion. For projects with workflows that are triggered by events, this could also delay the progression of those workflows. The engineering team is aware of the cause and is working to fix it. Next update by 4:30pm Pacific Time.
Jun 25, 15:31 PDT
All ingestion services continue to operate without any delay, so we're going to close out this incident. Thank you for your patience as we worked through this issue today. If you have any questions or concerns you can reach out to us at firstname.lastname@example.org
Jun 25, 11:43 PDT
There are no more data ingestion delays. We will continue to monitor to make sure that this issue does not recur.
Jun 25, 11:04 PDT
The ingestion delay has cleared up in most areas and it remains for only a very small subset of customers now. Next update by 11:00am PDT.
Jun 25, 10:07 PDT
The underlying issue has been resolved, and our engineering team is working on mitigating the remaining lag. Next update at 10:00am PDT
Jun 25, 08:58 PDT
All data ingestion services are currently experiencing lag, which is impacting event tracking, user updates, and CSV uploads. Our engineering team is working to resolve this issue. Next update at 9:00am PDT.
Jun 25, 08:00 PDT
Our API has stabilized and we are seeing no further issues. Thank you for your patience as we worked through this today. If you have any questions or concerns please contact us at email@example.com.
Jun 20, 18:10 PDT
Our /user/update API is back to a normal state. We will continue to monitor to ensure there are no further interruptions. Next Update by 6PM PST.
Jun 20, 17:25 PDT
We are experiencing high error rate and timeouts with our /users/update API endpoint. Our engineers are working diligently to resolve this issue. Next update by 6PM PST.
Jun 20, 17:02 PDT
The downstream infrastructure issues have been resolved and all API endpoints are ingesting properly. We will endeavor to identify what API calls might have been impacted during the outage and will be in touch with any customers associated with those calls. Thank you for your patience.
Jun 19, 17:55 PDT
This afternoon from approximately 4:05pm Pacific Time until 4:25pm, calls to the Bulk API endpoint, as well as calls to ListSubscribe and ListUnsubscribe, were failing, due to a downstream issue with our message processing infrastructure. This would also have impacted list uploads. The situation has now been mitigated, and depending upon the client-side retry logic, calls may yet process successfully. We are working to identify a list of calls that failed and connect with impacted customers. We are also continuing to monitor the infrastructure which was responsible to ensure it does not re-occur.
Next update by 6pm Pacific Time.
Jun 19, 16:54 PDT
The email preview tool is working properly and we are marking this incident as resolved. If you have any questions or concerns please reach out to us at firstname.lastname@example.org.
Jun 14, 14:01 PDT
We are continuing to investigate this issue.
Jun 14, 13:06 PDT
We have filtered out a subset of email previews and are continuing to work with our downstream rendering partner to identify the root cause. If you have any questions please reach out to your CSM or email@example.com. The next update will be before 2 PM Pacific Time.
Jun 14, 13:02 PDT
We're currently investigating an issue related to the rendering of email previews, and are working with our downstream rendering partner to determine the scope and root cause. For now, we're going to turn off the preview function. We'll have an update on this by 1pm Pacific Time.
Jun 14, 12:00 PDT