Codeship Pro, Delayed builds for projects on the "Small" instance size
Incident Report for Codeship

A spike in builds using the "small" instance type exhausted our build machine capacity. As our build machine management system scaled up to increase demand, the system encountered errors from an unoptimized backoff algorithm that prevented machines from being allocated to builds correctly. A fix to the algorithm has been made and we took manual steps to stabilize the system. Builds on "small" instances are running as expected, and Codeship regrets the service disruption.

Posted 4 months ago. Aug 09, 2017 - 15:25 UTC

Resolved
This incident is now resolved. We'll keep monitoring build VM provisioning closely, but build allocation times are within their normal range.
Posted 4 months ago. Aug 09, 2017 - 09:36 UTC
Monitoring
Our engineering team has taken steps to remedy this and we're now monitoring performance closely.
Posted 4 months ago. Aug 09, 2017 - 08:51 UTC
Investigating
We're looking into an issue with provisioning of build instances for the "Small" instance size causing delays for projects using those instances. Projects using a different instance size are not affected by this incident.
Posted 4 months ago. Aug 09, 2017 - 08:38 UTC