API instability

Incident Report for Iterable

Resolved

From 3:49PM UTC to 4:39PM UTC, 5:47PM UTC to 6:06PM UTC, our API machines suffered instability where you may've seen high latency or increased 5XX error codes.

This was caused by a sudden influx of API calls that overloaded the user database, causing a high number of threads to be in a blocked state. As a result, this impacted other API calls, like email/target that does not require a database lookup.

We've disabled the API key responsible for the influx of calls and we're working on an update to the API to reduce the number of database calls for the API.

We're looking into the putting our API behind the Amazon API Gateway where we can route APIs that require low latency to dedicated machines.

Posted Nov 15, 2016 - 14:48 PST