A spike in builds using the "small" instance type exhausted our build machine capacity. As our build machine management system scaled up to increase demand, the system encountered errors from an unoptimized backoff algorithm that prevented machines from being allocated to builds correctly. A fix to the algorithm has been made and we took manual steps to stabilize the system. Builds on "small" instances are running as expected, and Codeship regrets the service disruption.
Posted Aug 09, 2017 - 15:25 UTC
Resolved
This incident is now resolved. We'll keep monitoring build VM provisioning closely, but build allocation times are within their normal range.
Posted Aug 09, 2017 - 09:36 UTC
Monitoring
Our engineering team has taken steps to remedy this and we're now monitoring performance closely.
Posted Aug 09, 2017 - 08:51 UTC
Investigating
We're looking into an issue with provisioning of build instances for the "Small" instance size causing delays for projects using those instances. Projects using a different instance size are not affected by this incident.