Update - Our engineering team has continued to monitor the fix and we have seen no further issues with Journey entrances and processing. Total impact time was approximately from 5 am PDT until 8:36 am PDT. During this time no users were dropped, only delayed on their Journey entrances/ triggers. We will continue to monitor the health of Journey ingestion and processing before resolving this incident.
Dec 01, 2022 - 09:51 PST
Monitoring - Our team has identified the cause for the memory increase and applied a fix. The error rates are decreasing and the system is catching up. Operation has improved for workflow processing and we are currently monitoring the memory utilization. The next update at 9:30am PT or earlier.
Dec 01, 2022 - 09:02 PST
Investigating - As of approximately 7am PT due to workflow memory issues there has been a degradation in workflow processing across all customers. Our engineering team is investigating the root cause while allocating resources to reduce customer impact.
Next update at 9am PT or sooner.
Dec 01, 2022 - 08:24 PST