All Systems Operational
Web Application ? Operational
API ? Operational
Email Sending ? Operational
Email Links ? Operational
Workflows ? Operational
Push Sending Operational
SMS Sending Operational
System Webhooks Operational
Analytics Processing ? Operational
List Upload ? Operational
User Updates ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Web App Latency
Fetching
Links API Success Rate ?
Fetching
Links Latency
Fetching
Message Sending API Success Rate ?
Fetching
Message Sending API Latency
Fetching
User Update and Track API Latency ?
Fetching
Past Incidents
Jul 21, 2019

No incidents reported today.

Jul 20, 2019

No incidents reported.

Jul 19, 2019

No incidents reported.

Jul 18, 2019
Resolved - We have resolved the ingestion lag and all systems have now fully-caught up at this point. Thank you for your patience. If you have any concerns please contact us at support@iterable.com.
Jul 18, 18:01 PDT
Identified - Our team continues to work on a resolution. Ingestion lag as decreased and is improving both for custom events and api updates. Api error rates have decreased as well. Next Update will be at 6 pm PDT
Jul 18, 16:14 PDT
Investigating - Currently a subset of customers are experience delays in data ingestion for custom events and user api updates. We are also investigating elevated api error rates. Next update will be at 4 pm PDT
Jul 18, 14:55 PDT
Resolved - We have been fully caught up for some time now and are going to close this incident out. If you have any concerns please contact us at support@iterable.com.
Jul 18, 04:43 PDT
Update - Data ingestion is now fully caught up and we are going to monitor the queues for a little while longer before closing this incident out. Next update will be before 6 AM Pacific time.
Jul 18, 04:08 PDT
Monitoring - We have implemented changes that will solve the root cause of the issue. We are still experiencing data ingestion lag and are monitoring all queues until we see them return back to a normal state. The next update will be before 4 AM Pacific time.
Jul 18, 03:04 PDT
Identified - One of our server clusters is currently experiencing data ingestion lag of up to an hour. The engineering team has identified what caused the issue and is currently working on a fix. They are also working on reducing the backlog of data ingestion. The next update will be before 3 AM Pacific Time.
Jul 18, 02:00 PDT
Jul 17, 2019
Resolved - Data ingestion is now fully-caught up at this point. Thank you for your patience. If you have any concerns please contact us at support@iterable.com
Jul 17, 10:46 PDT
Investigating - One of our server clusters is currently experiencing data ingestion lag of up to 30 minutes. This is impacting approximately 40% of customers. The engineering team is working on addressing the backlog right now. Impacted customers will see ingestion of API data and CSV uploads delayed. Next update by 10:40am Pacific Time.
Jul 17, 09:39 PDT
Jul 16, 2019

No incidents reported.

Jul 15, 2019

No incidents reported.

Jul 14, 2019

No incidents reported.

Jul 13, 2019

No incidents reported.

Jul 12, 2019

No incidents reported.

Jul 11, 2019

No incidents reported.

Jul 10, 2019

No incidents reported.

Jul 9, 2019
Resolved - This incident has been resolved.
Jul 9, 16:03 PDT
Update - We have not encountered any further issues with workflow delays, so we are now closing this incident. Thank you for your patience if you were adversely affected, and if there are any questions or concerns, please open a ticket by emailing support@iterable.com
Jul 9, 10:14 PDT
Monitoring - Workflow queues have now cleared and we are back to operating as normal. We will continue to monitor this through the night. Further actions will be taken tomorrow to continue addressing the root cause of this. Next update by 8:30am Pacific Time, unless more information is identified prior.
Jul 8, 19:37 PDT
Update - Workflow queues for impacted customers continue to improve, reducing by about 35% in the last hour. We are continuing to work on making adjustments to improve throughput. Next update by 7:30pm Pacific Time.
Jul 8, 17:28 PDT
Update - We are down to about ten customers still seeing workflow backups. Those queues are recovering and the engineering team is continuing to work on options for speeding up that recovery. Next update by 5pm Pacific Time.
Jul 8, 15:32 PDT
Update - Most workflow queues are now processing successfully, although there is still some residual backup. We're continuing to work on rectifying a root cause of some of the backup, to ensure the situation does not recur. Next update by 3:30pm Pacific Time.
Jul 8, 14:00 PDT
Identified - We've made some good progress in the last 30 minutes, hence the slight delay in updates. The workflow queue backups are continuing to process, although there are a lot of events to get through. We have identified the cause of the workflows that were delayed for other reasons (as mentioned in the previous update) and are working on deploying new code to fix that root cause. Next update by 1:45pm Pacific Time.
Jul 8, 12:19 PDT
Update - The team is making progress on identifying root causes for the delayed workflows. Most workflow delays are due to yesterday's ingestion issues and the system is making progress on getting through that backlog. There are a few workflows that are delayed for other reasons, and it is those situations which require additional investigation, which is ongoing. Next update by 12pm Pacific Time.
Jul 8, 10:34 PDT
Investigating - We're seeing some lingering issues from yesterday's data ingestion delays that are causing workflow processing delays for some customers. The engineering team is working on resolving the backups right now. We'll post another update by 10:30am Pacific Time.
Jul 8, 09:12 PDT
Jul 7, 2019
Resolved - We are comfortable now setting this issue to resolved as we no longer are reporting any latency. If you see any issues you believe may be related please reach out to support@iterable.com.
Jul 7, 19:00 PDT
Update - Thank you for everyone's patience while the team was hard at work investigating the root cause of the issue. We are now seeing no latency being reported, and our machines are showing signs of normal health. We will continue monitoring for the time being, and move to resolved once we feel confident. If you see any issues please reach out to support@iterable.com.
Jul 7, 18:01 PDT
Monitoring - We are now showing signs of our machines returning to a normal state with no latency issues. We are still continuing to monitor for the time being, though we are seeing increasing improvements in terms of speed. Next update at 9PM EDT.
Jul 7, 17:03 PDT
Update - We are continuing work on the latency issue, and are starting to see progress in terms of restoring to a normal state. We will continue to monitor progress, and post updates as they become available. Next update will be 8PM EDT.
Jul 7, 16:02 PDT
Update - The team is hard at work exploring options that can be taken to help with the latency issues, and at this time investigation is still ongoing. Next update at 7PM EDT.
Jul 7, 15:03 PDT
Update - Investigation continues on latency issues. Next update will be at 6PM EDT
Jul 7, 14:03 PDT
Update - Investigation continues on latency issues. Next update will be at 5PM EDT
Jul 7, 13:03 PDT
Update - We are still investigating the root cause for the latency issues. Next update will be at 4PM EDT.
Jul 7, 12:01 PDT
Update - We are still investigating the root cause for the latency issues, but have made some progress since the last update. Next update at 3PM EDT.
Jul 7, 11:00 PDT
Update - We are still in the process of investigating the cause of the latency issues, and our team is currently exploring options for returning to a normal state. We have identified that the affected areas are only a subset of machines, and is not affecting the entire platform. Next update at 2PM PDT
Jul 7, 10:07 PDT
Investigating - We have identified latency occurring with a subset of our machines, and are experiencing latency up to 1 hour. Our team is currently looking into the issue and working on resolving the latency to a normal state. Next update will be 1PM EDT
Jul 7, 08:50 PDT