Ingestion on c11 stopped

Incident Report for Iterable

Resolved

c11 has fully recovered and the queue has cleared.
Posted Mar 22, 2024 - 14:36 PDT

Update

We have determined the impact was an increase in failed requests to user update API and only impacted a small subset of customers. No sends or triggering of campaigns were dropped, although some may have been delayed. Customers should try any failed requests again. We will continue monitoring the cluster health before closing out this incident.
Posted Mar 22, 2024 - 14:00 PDT

Update

We have determined the impact was an increase in failed requests to user update API and only impacted a small subset of customers. No sends or triggering of campaigns were dropped, although some may have been delayed. Customers should try any failed requests again. We will continue monitoring the cluster health before closing out this incident.
Posted Mar 22, 2024 - 13:58 PDT

Investigating

Starting at 12:15PT, C11 is experiencing performance issue. Customer may experience API requests fail such as triggering campaigns and user profile updates. Scheduled messages are still queued to send, but may be delayed. We are showing improvement. Please retry any failed API requests. Next update by 2PT.
Posted Mar 22, 2024 - 13:17 PDT
This incident affected: Cluster 11 (Email Sends, Workflow Processing, Push Sends, SMS Sends, User Updates, List Uploads, User Deletions).