We continue to see no new impact related to the memory issues earlier in the day. Our engineers have uncovered one side effect in which in certain limited cases it appears as though users are stuck in a journey. We have confirmed that in these cases it is an issue with the UI, and that the users did in fact progress. We have initiated efforts to remediate the incorrect counts of stuck users, and we expect that process to be completed tomorrow (Friday). For now, we will go ahead and close out this incident. Thank you for your patience - please reach out to us at support@iterable.com if you have any further questions.
Posted Dec 01, 2022 - 19:15 PST
Update
Our engineering team has continued to monitor the fix and we have seen no further issues with Journey entrances and processing. Total impact time was approximately from 5 am PDT until 8:36 am PDT. During this time no users were dropped, only delayed on their Journey entrances/ triggers. We will continue to monitor the health of Journey ingestion and processing before resolving this incident.
Posted Dec 01, 2022 - 09:51 PST
Monitoring
Our team has identified the cause for the memory increase and applied a fix. The error rates are decreasing and the system is catching up. Operation has improved for workflow processing and we are currently monitoring the memory utilization. The next update at 9:30am PT or earlier.
Posted Dec 01, 2022 - 09:02 PST
Investigating
As of approximately 7am PT due to workflow memory issues there has been a degradation in workflow processing across all customers. Our engineering team is investigating the root cause while allocating resources to reduce customer impact.