As of 4:16 PM PST, all affected messages have been successfully processed, and the incident is now resolved. Our engineering team has confirmed that all campaigns have been reviewed, and there are no outstanding issues.
The 3rd party vendor's recovery at 8:07 AM PT resolved the initial sending issue, and no further delays or interruptions are expected.
We appreciate your patience during this time and apologize for any inconvenience caused. If you have any questions or need further assistance, please reach out to our support team at support@iterable.com
Posted Oct 15, 2024 - 14:32 PDT
Update
Engineering is continuing to work on identifying alternative methods to determine affected campaigns. We will be reaching out to impacted customers to confirm whether redelivery is necessary. The 3rd party vendor recovered at 8:07 AM PT, and there are no ongoing issues with email sends.
Posted Oct 15, 2024 - 12:22 PDT
Update
Engineering is identifying alternative methods to identify affected campaigns and will contact customers to confirm whether redelivery is needed. Additionally, the 3rd party vendor recovered as of 8:07 am PT, and there are no issues with send. Next update at 12 pm PT.
Posted Oct 15, 2024 - 11:12 PDT
Update
We are continuing to monitor for any further issues.
Posted Oct 15, 2024 - 11:11 PDT
Update
Engineering is identifying alternative methods to identify affected campaigns and will contact customers to confirm whether redelivery is needed. Additionally, 3rd party vendor recovered as of 8:07 am PT and there are no issues with send. Next update at 12 pm PT.
Posted Oct 15, 2024 - 11:05 PDT
Update
The fix from our third party vendor has been stable and we're seeing email campaigns are sending successfully again. We are currently working to resend the emails that were impacted during the impact window.
Next update will be at 11:00 am pst or sooner.
Posted Oct 15, 2024 - 09:49 PDT
Monitoring
Starting around 7:27 AM PST, Iterable engineers began receiving alerts about elevated error rates on a 3rd-party ESP provider's API endpoints.
Iterable has been working directly with our 3rd party vendor and they have released a fix and as of 8:07 AM PST we are seeing successful email sends.
Customers using this ESP may experience issues impacting email sends, including elevated email campaign send skips during this impact window
Iterable Engineer will continue monitoring the issue. If you have any questions please reach out to support@iterable.com
Posted Oct 15, 2024 - 09:06 PDT
This incident affected: Global Campaign Sends and Global Proof Sends.