There was an issue with a third party database connection pool library that surfaces under heavy load. In the short term, we're adding additional alerting so that we learn about the issue before it has customer impact. We have also added more servers to increase workflow throughput. In the medium term, we're upgrading the third party library which caused the issue. We apologize to the clients that were impacted by this incident.
Posted Jul 12, 2017 - 13:27 PDT
Resolved
Our workflow processing servers stopped processing causing the queues for certain projects to be backed up. The incident in resolved and we've added more servers to prevent this moving forward.
Posted Jul 11, 2017 - 18:06 PDT
Investigating
We are currently experiencing slowdowns in back-end processing that may be impacting workflow processing for certain projects. We are investigating the root cause now.