Queued Builds on Codeship Basic
Incident Report for CodeShip
Postmortem

Yesterday (19th June 2017) we experienced issues with provisioning a certain set of AWS EC2 instance types on the Codeship Basic platform. This resulted in lack of capacity to process Codeship Basic builds and delays in processing. It appears that AWS was having capacity issues with particular instance types in the us-east-1 region, which lead to errors when our system attempted to scale up and launch new instances. To address this issue we first cast our net wider across the availability zones in the us-east-1 region, in the hopes of obtaining more resources. Once we were clear the issue persisted we switched to an instance type that was reporting more availability. We manually started lots of instances and scaled up our workers to increase our build throughput and reduce any further delays.

Posted Jun 20, 2017 - 18:01 UTC

Resolved
This incident is now resolved. Depending on account limits there might still be some queued builds, but those will be process as fast as possible. Any new incoming builds are process right away (unless account limits cause the build to get queued).
Posted Jun 19, 2017 - 13:19 UTC
Monitoring
We've mostly caught up with the demand and are working through the remaining queued builds. We should be back operating normally soon.
Posted Jun 19, 2017 - 12:49 UTC
Update
We're adding new capacity to our build pool as fast as possible to deal with both the backlog of waiting builds as well as the increased load during US working hours.
Posted Jun 19, 2017 - 12:19 UTC
Identified
We traced the issue back to capacity issues with AWS and are looking at possible workarounds. We're further spawning more instances and see a slight decrease in queue times for builds.
Posted Jun 19, 2017 - 11:42 UTC
Update
We're starting additional instances to deal with the queued builds and are further looking into this.
Posted Jun 19, 2017 - 11:16 UTC
Investigating
We're looking into an issue with longer than usual queue times for builds on Codeship Basic.
Posted Jun 19, 2017 - 10:54 UTC