The original journey performance issues that triggered this incident have now been resolved. As mentioned in previous updates, there may be customers that have experienced users stuck in journey tiles as an effect of this incident. For these customers, our Customer Success team will be working with you separately to address these users on a case by case basis. Because the original impact is resolved, this incident will be closed, and a root cause analysis will be in progress. If you experience further journey performance issues, please reach out the Support team.
Posted Jun 29, 2023 - 18:12 PDT
Update
The journeys backlog for all clusters has been caught up since around 12:50pm PDT. A possible effect during the impact period is users getting stuck in journey tiles. The Engineering team is still working on options to fix these users. An update will be provided when there is further progress.
However, if you would like to clear your users out of their journeys from where they became stuck, you may be able to do so now. Please work with your CSM or with Support to see if action can be taken for your case.
Posted Jun 28, 2023 - 17:35 PDT
Update
Most clusters are caught up following the deployed update. The team is investigating for other effects during the problem period. We will continue to monitor journey performance and working with impacted customers.
Posted Jun 28, 2023 - 07:45 PDT
Update
The Engineering team continues to monitor all clusters. We are still seeing a downward trend, indicating backlog is depleting. Next update will be at 9 AM PDT or sooner.
Posted Jun 28, 2023 - 03:56 PDT
Update
As our Engineering team continue to monitor, we are seeing all clusters continue to catch up.
Posted Jun 28, 2023 - 03:10 PDT
Update
Our engineering team are continuing to monitor the performance and majority of clusters are catching up. The team is still working on some remaining performance degradation on c23. The team will continue to monitor the situation. The next update will be at 3 AM PDT or sooner.
Posted Jun 28, 2023 - 01:57 PDT
Monitoring
Our Engineering team have been working tirelessly today. The team has deployed a possible fix to address a bottleneck in processing the backlog of journey events. The team will continue to monitor throughout the night. The next update will be at 1 AM PDT or sooner.
Posted Jun 27, 2023 - 23:16 PDT
Update
Our Engineering team have identified the root cause. We are working with our 3rd party downstream provider to resolve the underlying issue. The next update will be at 11 PM PDT or sooner.
Posted Jun 27, 2023 - 21:11 PDT
Update
Our engineering team is still reporting journey latency. The network configurations have been updated. They are continuing to work with our downstream vendors. Next Update at 9 PM PDT or sooner.
Posted Jun 27, 2023 - 19:13 PDT
Identified
Our engineering team is still reporting journey latency due to possible mitigation efforts. They are continuing to work with our downstream vendors as well as adding additional logging to investigate the issue. Next Update at 7 PM PDT or sooner.
Posted Jun 27, 2023 - 18:02 PDT
Update
We are continuing to monitor the applied mitigations and work through the existing backlog of Journey events. Next Update at 6 PM PDT or sooner if the backlog has caught up before then.
Posted Jun 27, 2023 - 16:09 PDT
Monitoring
After working with our downstream vendors our engineering team has taken multiple mitigation steps and implemented a fix. Since the fix was applied, we are seeing journey event processing rapidly increase as our system works through the large backlog of Journey events across all the impacted clusters. We are continuing to monitor and are hoping to be caught up within the next few hours or sooner as the additional resources allocated help with processing. As we continue to process the large amount of events, customers should start to see some journeys pick up their processing speed and the associated actions within impacted Journeys should start to process through (triggers, delay nodes, filters, send nodes). Our next update will be at 4 PM PDT or sooner.
Posted Jun 27, 2023 - 15:06 PDT
Identified
We are continuing to investigate underlying causes and have been working with multiple 3rd party downstream providers on mitigations and allocating more resources to increase the processing power and reduce the current Journey processing delays. We have identified a possible fix and are in the process of implementing it. Customers impacted across multiple clusters will still experience delays on all workflow trigger actions (custom events, API triggers, scheduled list triggers) and also see delays in workflow node actions and send tiles that are part of the Journeys. Please note that all events, triggers, and sends associated with impacted Journeys are queued up and will eventually process as no data is being dropped. Next update will be at 3 PM PDT or sooner
Posted Jun 27, 2023 - 14:09 PDT
Update
Our engineering team is working through possible solutions and remediation steps for all impacted accounts. A fix has not been implemented. Customers can still expect delays in journey triggers, users processing within workflows, and messages being sent from journey send tiles. No journey or user data is being dropped, only delayed. Next update at 2PM PST or sooner.
Posted Jun 27, 2023 - 12:58 PDT
Update
Our engineering team has identified the cause of journey latency and is deploying new resources. Customers will still experience delays in journey triggers, users processing within workflows, and messages being sent from journey send tiles. No journey or user data is being dropped, only delayed. Our team is actively working on full remediation steps. Next update at 1PM PST or sooner.
Posted Jun 27, 2023 - 12:01 PDT
Update
Our engineering team has added additional resources to mitigate delays on select journeys. Customers may still see delays in journey triggers, users processing within workflows, and messages being sent from journey send tiles. No journey or user data is being dropped, only delayed. Our team is actively working on full remediation steps. Next update at 12PM PST or sooner.
Posted Jun 27, 2023 - 11:04 PDT
Update
Our engineering team is actively working on remediation steps for this issue and adding additional resources to improve journey processing. Impacted customers may see delays in journey triggers, users processing within workflows, and messages being sent from journey send tiles. No journey or user data is being dropped, only delayed. Next update by 11 am PST or sooner.
Posted Jun 27, 2023 - 09:49 PDT
Update
We are continuing to investigate this issue.
Posted Jun 27, 2023 - 08:56 PDT
Investigating
Iterable on-call engineers are currently investigating reports of Journey's processing users slower than expected across all clusters. Next update will be at 9:45 am PST.