Stuck Builds on Codeship Basic
Incident Report for CodeShip
Postmortem

We published a post mortem for this incident over at our blog. Please see Postmortem: Codeship Basic Platform Build Processing Incidents for more details.

TL;DR A change to our Codeship Basic machine pool configuration uncovered an issue with build container created_at timestamps, leading to pipeline failures due to programmatic termination of long-running processes by the Codeship Basic build machine agent.

Posted Jun 16, 2017 - 10:20 UTC

Resolved
Builds are processing as planned. We keep monitoring the build queues and will make further changes to improve stability.
Posted Jun 05, 2017 - 21:31 UTC
Monitoring
We've resolved the issue and are monitoring our systems. There still might be builds affected by this and we are identifying those and taking steps to unstuck those builds.
Posted Jun 05, 2017 - 18:16 UTC
Update
We are taking necessary steps to keep impact on users as minimal as possible and are further looking into the root cause for this.
Posted Jun 05, 2017 - 15:58 UTC
Investigating
We are seeing builds stuck in queue erroneously and are investigating further.
Posted Jun 05, 2017 - 14:01 UTC