Codeship Pro, Delayed builds for projects on the "Small" instance size
Incident Report for Codeship

A spike in builds using the "small" instance type exhausted our build machine capacity. As our build machine management system scaled up to increase demand, the system encountered errors from an unoptimized backoff algorithm that prevented machines from being allocated to builds correctly. A fix to the algorithm has been made and we took manual steps to stabilize the system. Builds on "small" instances are running as expected, and Codeship regrets the service disruption.

Posted 11 months ago. Aug 09, 2017 - 15:25 UTC

This incident is now resolved. We'll keep monitoring build VM provisioning closely, but build allocation times are within their normal range.
Posted 11 months ago. Aug 09, 2017 - 09:36 UTC
Our engineering team has taken steps to remedy this and we're now monitoring performance closely.
Posted 11 months ago. Aug 09, 2017 - 08:51 UTC
We're looking into an issue with provisioning of build instances for the "Small" instance size causing delays for projects using those instances. Projects using a different instance size are not affected by this incident.
Posted 11 months ago. Aug 09, 2017 - 08:38 UTC