We're closing out this API incident, because our engineers have determined the root cause. A minor change to how one piece of our logging systems was configured caused threads to get blocked on our API servers. We have reverted to the previous configuration. Please reach out to support@iterable.com if you have any questions.
Posted Nov 01, 2018 - 19:58 PDT
Monitoring
API endpoints are once again functioning properly. We've been able to gather a lot of data surrounding the higher-than-normal error rates this morning and last night, and the team is analyzing this to identify a root cause or causes.
Posted Nov 01, 2018 - 12:10 PDT
Investigating
We're seeing some of our API machines still behaving poorly, so we've moved back to "investigating". The engineering team is working on narrowing down a root cause right now. Overall impact appears to be minimal for the time being, so we'll send out another update when we have more detail on root cause, or by noon Pacific Time, whichever is soonest.
Posted Nov 01, 2018 - 10:10 PDT
Update
API requests are still behaving correctly and we expect to move this incident to a "resolved" status shortly.
Posted Nov 01, 2018 - 08:53 PDT
Monitoring
API requests have returned to normal and our on-call engineers are currently monitoring. We will continue to identify the root cause of this and will provide an update by 9:00am Pacific Time.
Posted Oct 31, 2018 - 23:14 PDT
Investigating
We're investigating another case of API requests returning higher than usual errors. We've identified that these errors specifically have to do with the /users/bulkUpdate API endpoint. Our on-call engineers are working on a potential solution and we will provide an update by 11:45pm Pacific Time.
Posted Oct 31, 2018 - 22:45 PDT
Update
Our on-call engineers are continuing to monitor the API requests. We have yet to identify a root cause of the unusually high errors, but we will continue to investigate and will provide an update by 9am Pacific Time.
Posted Oct 31, 2018 - 22:03 PDT
Monitoring
API requests are no longer returning higher than usual errors. We're monitoring the system and continuing to identify the root cause. Next update at 10pm or when the incident moves to a new status.
Posted Oct 31, 2018 - 20:41 PDT
Update
We're still working on finding a root cause. We will update again by 9:00pm.
Posted Oct 31, 2018 - 20:02 PDT
Investigating
We're investigating API requests returning higher than usual errors. We are working on determining the scope at this point and will provide an update by 8:00pm Pacific Time.