Resolved
We have fully recovered from this incident and are marking it resolved. If you have any further questions please reach out to support@iterable.com
Posted Jul 19, 2024 - 12:10 PDT
Update
As of now we have completely caught up on ingestion lag with all services returning to normal. We will continue to monitor performance with that next update at 12 PM PT or sooner.
Posted Jul 19, 2024 - 11:00 PDT
Update
We are continuing to monitor for any further issues.
Posted Jul 19, 2024 - 10:38 PDT
Monitoring
Web app and API endpoints have completely recovered at this point. However, there are still a subset of customers that may be experiencing an ingestion lag that is currently draining. These customers may still be seeing delays in user updates, event calls, and event triggered journeys. We are continuing to monitor this and will provide our next update at 11 AM PT or sooner.
Posted Jul 19, 2024 - 10:10 PDT
Identified
Our Engineers have identified the root cause of the issue and are actively working on deploying a fix. Currently our API endpoints have recovered, but in the meantime, customers may still be experiencing slowness in scheduled and triggered Journeys, scheduled and triggered campaign sends, custom events, user updates, and more. Customers may also be experiencing webApp performance degradation as well including segmentation, list uploads, and viewing campaign details. While the fix is being deployed, to clarify, this issue is specifically impacting All clusters numbered 100+. Our next update will be at 10:00 AM PT or sooner.
Posted Jul 19, 2024 - 09:16 PDT
Update
We are continuing to investigate this issue.
Posted Jul 19, 2024 - 08:43 PDT
Investigating
Beginning around 6:20 AM PST we were alerted to a spike in API errors across multiple endpoints impacting a number of specific customer clusters. All clusters numbered 100+ may be experiencing a spike in 5xx API errors across all endpoints. This may impact areas of the app such as scheduled and triggered Journeys, scheduled and triggered campaign sends, custom events, user updates, and more. Customers may also be experiencing webApp performance degradation as well including segmentation, list uploads, and viewing campaign details.
Our engineering team is continuing work on identifying the underlying cause and exploring remediation options. Next update will be at 9 AM PST or sooner. If you have questions please reach out to support@iterable.com
Posted Jul 19, 2024 - 08:37 PDT
This incident affected: Cluster 100 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 101 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 102 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 103 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 104 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 105 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 106 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 107 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 108 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 109 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 110 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 111 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 112 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 113 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 114 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 115 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 116 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 117 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 118 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 119 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 120 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 121 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 122 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 123 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 124 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 125 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 126 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 127 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 128 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 129 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), Cluster 130 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Updates, User Deletions), and Global Links.