Resolved -
Our Engineering team has confirmed at 12:43PM PDT that all scheduled campaigns, scheduled journey triggers, sends for scheduled campaigns, and export list calls are behaving normally and are stable. This incident is now resolved.
Mar 18, 13:11 PDT
Update -
Engineering is no longer seeing any delays in scheduled campaigns, scheduled journey triggers, partial sends for a subset of scheduled campaigns, and export list calls. We will continue to monitor this fix across all our systems and ensure there are no errors for the next day. If we continue to see no errors, we will resolve this issue tomorrow at 12 PM PDT.
We will provide another update by tomorrow at 12 PM PDT or sooner if we have significant updates.
Mar 17, 14:57 PDT
Monitoring -
Engineering has reviewed the changes it pushed through this weekend and has not seen any delays in scheduled campaigns, scheduled journey triggers, partial sends for a subset of scheduled campaigns, and export list calls. We are currently monitoring the fix across all our systems to ensure we did not miss any services. While the fix is in, it might also take some time for the changes to fully propagate.
We will provide another update by 3PM PDT or sooner if we have significant updates.
Mar 17, 09:48 PDT
Update -
Engineering continues to monitor the changes we made this weekend. So far, our monitoring has indicated significant performance improvements. The next step is to observe and confirm that the changes have mitigated the delays in campaign sends, journey triggers, and processing after our system has received increased traffic.
The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We will provide another update tomorrow before 10 AM PDT or sooner if we have significant updates.
Mar 16, 15:05 PDT
Update -
Engineering is actively monitoring the fix implemented last night. While initial metrics show signs of improvement, we are still assessing whether delays in campaign sends, journey triggers, and processing have been fully resolved.
To ensure a thorough resolution, we will continue tracking the impact of this fix while exploring additional solutions over the weekend. The scope of the issue remains unchanged, affecting fewer than 1% of API calls across our customer base.
Our next update will be provided by 3 PM PDT tomorrow, or sooner if we have significant developments.
Mar 15, 14:44 PDT
Update -
Engineering is still investigating a complete fix, and are monitoring changes we put in last night to see if it mitigates this issue. We are seeing performance improvements across delays in campaign sends, journey triggers, and processing. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We will provide another update before 3 PM PDT or sooner if we have significant updates.
Mar 15, 09:40 PDT
Update -
Engineering is still investigating a complete fix, but is still seeing performance improvements across delays in campaign sends, journey triggers, and processing. We are continuing to work on rolling out improvements and solutions. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We will provide another update tomorrow by 10 AM PDT or sooner if we have significant updates.
Mar 14, 16:05 PDT
Update -
Engineering continues to pursue different fixes. Performance for our customers has been partially remediated and improved. Customers should be experiencing a reduction in delays in campaign sends, journey triggers, and processing. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base. We will provide another update by 5 PM PDT or sooner if we have significant updates.
Mar 14, 13:17 PDT
Update -
Engineering has implemented remediation steps, but they have not resolved the issue. The team is actively exploring alternative solutions and evaluating their effectiveness. The scope of the issue remains unchanged, affecting less than 1% of calls across our customer base.
We appreciate your patience and will provide the next update by 1 PM PDT or sooner if we have significant updates
Mar 14, 09:26 PDT
Update -
Engineering continues to work on remediation steps and is testing fixes on individual services to minimize customer impact. The scope of the issue remains limited, affecting less than 1% of calls across our customer base.
Customers may experience sporadic errors or failures in the following areas:
- Scheduled campaigns: Delays in starting or finishing
- Scheduled journey triggers
- Partial sends for a subset of scheduled campaigns
- Export list calls
We appreciate your patience as we work toward a resolution. The next update will be provided by 10 AM PDT tomorrow or sooner if significant progress is made.
Mar 13, 14:17 PDT
Update -
Engineering is continuing to pursue remediation steps. They are still investigating different solutions and attempting to test fixes on certain services individually to help ensure that more significant impact is not felt for our customers. We will provide the next update by 3 PM PDT or sooner.
Mar 13, 12:20 PDT
Update -
Engineering has continued working through its remediation steps. Currently, it is focusing on testing a fix on certain services individually to help ensure that our customers do not feel a greater impact. We will provide the next update by 12 PM PDT or sooner.
Mar 13, 09:38 PDT
Update -
The engineering team is continuing to investigate reduced performance with campaign sends, scheduled journeys, and exports. This issue occurs sporadically and does not affect every campaign or journey. If you think you're affected by this issue, please contact support@iterable.com. We will provide the next update by 10 AM PDT tomorrow or before.
Mar 12, 16:49 PDT
Update -
Investigating - The engineering team is still investigating reports of customers experiencing degraded performance with campaign sends, scheduled journeys, and exports. If you believe you're seeing this impact, please contact support@iterable.com. The next update will be before 5PM PST today.
Mar 12, 10:33 PDT
Investigating -
The engineering team has received reports of customers experiencing degraded performance with campaign sends, scheduled journeys, and exports. They are currently investigating the issue. If you believe you're seeing this impact, please contact support@iterable.com. The next update will be at 10:30 AM PST.
Mar 12, 08:45 PDT