Caching currently disabled for Codeship Pro builds
Incident Report for Codeship
Postmortem

Last week, we experienced some issues with builds on the Codeship Pro platform. This was due to the way we configure caches for builds. Project build caches are backed by Amazon ECR and we encountered an unforeseen limit in the number of repositories we can create. This led to builds not getting executed, while they waited for a repository to be created.

During the incident, we momentarily disabled caching to get builds moving. We then deleted a large number of unused repositories and re-enabled caching.

This was a temporary measure and we have identified a number of near and long-term actions to prevent this from ever impacting your builds again. This includes improvements into visibility of our service provider limits, garbage collection of unused repositories and automated handling of this error case to mitigate against complete halts in our workloads.

We apologize for the disruption to your builds and understand the impact of our service disruptions.

Posted 2 months ago. Jun 04, 2018 - 10:20 UTC

Resolved
This incident has been resolved.
Posted 3 months ago. May 30, 2018 - 21:14 UTC
Investigating
Regarding our recent Codeship Pro outage, our caching feature is currently offline.
Posted 3 months ago. May 30, 2018 - 20:21 UTC
This incident affected: Codeship Pro (Caching).