Degraded Back-end Processing Performance

Incident Report for Iterable

Postmortem

There was an issue with a third party database connection pool library that surfaces under heavy load. In the short term, we're adding additional alerting so that we learn about the issue before it has customer impact. We have also added more servers to increase workflow throughput. In the medium term, we're upgrading the third party library which caused the issue. We apologize to the clients that were impacted by this incident.

Posted Jul 12, 2017 - 13:27 PDT

Resolved

Our workflow processing servers stopped processing causing the queues for certain projects to be backed up. The incident in resolved and we've added more servers to prevent this moving forward.
Posted Jul 11, 2017 - 18:06 PDT

Investigating

We are currently experiencing slowdowns in back-end processing that may be impacting workflow processing for certain projects. We are investigating the root cause now.
Posted Jul 11, 2017 - 12:52 PDT