Stuck Builds on Codeship Basic

Incident Report for CodeShip

Postmortem

We published a post mortem for this incident over at our blog. Please see Postmortem: Codeship Basic Platform Build Processing Incidents for more details.

TL;DR A change to our Codeship Basic machine pool configuration uncovered an issue with build container created_at timestamps, leading to pipeline failures due to programmatic termination of long-running processes by the Codeship Basic build machine agent.

Posted Jun 16, 2017 - 10:20 UTC

Resolved

Builds are processing as planned. We keep monitoring the build queues and will make further changes to improve stability.
Posted Jun 05, 2017 - 21:31 UTC

Monitoring

We've resolved the issue and are monitoring our systems. There still might be builds affected by this and we are identifying those and taking steps to unstuck those builds.
Posted Jun 05, 2017 - 18:16 UTC

Update

We are taking necessary steps to keep impact on users as minimal as possible and are further looking into the root cause for this.
Posted Jun 05, 2017 - 15:58 UTC

Investigating

We are seeing builds stuck in queue erroneously and are investigating further.
Posted Jun 05, 2017 - 14:01 UTC