Stuck builds on CodeShip Pro

Incident Report for CodeShip

Postmortem

On Monday 15th October Pro customers may have experienced some of their builds getting stuck. This may have presented itself as build steps or logs halting mid-execution or not stopping properly when triggered to do so. The root cause of this issue was found to be a degradation in our database provider's network. We have begun the process to mitigate against network problems like these from impacting customers in the future. We sincerely apologize for any impact this may have had to your workflow.

Posted Oct 17, 2018 - 10:17 UTC

Resolved

This incident has been resolved.
Posted Oct 16, 2018 - 03:37 UTC

Update

Pro builds should be operating normally again. Contact support@codeship.com if you see any further issues.
Posted Oct 16, 2018 - 03:35 UTC

Monitoring

Database maintenance is complete and we are monitoring.
Posted Oct 16, 2018 - 00:23 UTC

Identified

We are performing database maintenance now to help resolve this issue.
Posted Oct 15, 2018 - 23:39 UTC

Update

We are seeing database connection timeouts that suggest network issues and are working with Heroku to further diagnose.
Posted Oct 15, 2018 - 19:57 UTC

Investigating

Investigating stuck builds on CodeShip Pro.
Posted Oct 15, 2018 - 17:58 UTC
This incident affected: CodeShip Pro (Builds).