Codeship Basic, Delays with build allocation
Incident Report for CodeShip
Postmortem

On August 9, 2017 from 9:29AM to 11:59AM EDT, Codeship Basic builds were delayed in processing with delays peaking at 36 minutes during the height of the backlog. Codeship users saw that their build pipelines would not start running immediately after they were created as there was contention for servers in our elastic build infrastructure. The root cause of this incident was a lack of capacity in our fleet of cloud compute instances to process the queue of builds in real-time. These capacity issues were caused by error responses from our cloud provider when requesting new instances to scale up our fleet. Other smaller issues were identified that contributed to making it more difficult to mitigate the impact such as machines being taken offline from aggressive health checks. Build queues and wait times are now back to normal. We apologize for the service disruption.

Posted Aug 15, 2017 - 18:37 UTC

Resolved
This incident is now resolved.
Posted Aug 09, 2017 - 17:44 UTC
Monitoring
Continuing to monitor as build processing has stabilized on our Basic service.
Posted Aug 09, 2017 - 15:13 UTC
Investigating
We're looking into delays with build processing on Codeship Basic. Builds run but are roughly 15 minutes delayed at the moment.
Posted Aug 09, 2017 - 13:30 UTC