Resolved -
We have fully recovered from this incident and are marking it resolved. If you have any further questions please reach out to support@iterable.com
Jul 19, 12:10 PDT
Update -
As of now we have completely caught up on ingestion lag with all services returning to normal. We will continue to monitor performance with that next update at 12 PM PT or sooner.
Jul 19, 11:00 PDT
Update -
We are continuing to monitor for any further issues.
Jul 19, 10:38 PDT
Monitoring -
Web app and API endpoints have completely recovered at this point. However, there are still a subset of customers that may be experiencing an ingestion lag that is currently draining. These customers may still be seeing delays in user updates, event calls, and event triggered journeys. We are continuing to monitor this and will provide our next update at 11 AM PT or sooner.
Jul 19, 10:10 PDT
Identified -
Our Engineers have identified the root cause of the issue and are actively working on deploying a fix. Currently our API endpoints have recovered, but in the meantime, customers may still be experiencing slowness in scheduled and triggered Journeys, scheduled and triggered campaign sends, custom events, user updates, and more. Customers may also be experiencing webApp performance degradation as well including segmentation, list uploads, and viewing campaign details. While the fix is being deployed, to clarify, this issue is specifically impacting All clusters numbered 100+. Our next update will be at 10:00 AM PT or sooner.
Jul 19, 09:16 PDT
Update -
We are continuing to investigate this issue.
Jul 19, 08:43 PDT
Investigating -
Beginning around 6:20 AM PST we were alerted to a spike in API errors across multiple endpoints impacting a number of specific customer clusters. All clusters numbered 100+ may be experiencing a spike in 5xx API errors across all endpoints. This may impact areas of the app such as scheduled and triggered Journeys, scheduled and triggered campaign sends, custom events, user updates, and more. Customers may also be experiencing webApp performance degradation as well including segmentation, list uploads, and viewing campaign details.
Our engineering team is continuing work on identifying the underlying cause and exploring remediation options. Next update will be at 9 AM PST or sooner. If you have questions please reach out to support@iterable.com
Jul 19, 08:37 PDT