Last week, we experienced some issues with builds on the Codeship Pro platform. This was due to the way we configure caches for builds. Project build caches are backed by Amazon ECR and we encountered an unforeseen limit in the number of repositories we can create. This led to builds not getting executed, while they waited for a repository to be created.

During the incident, we momentarily disabled caching to get builds moving. We then deleted a large number of unused repositories and re-enabled caching.

This was a temporary measure and we have identified a number of near and long-term actions to prevent this from ever impacting your builds again. This includes improvements into visibility of our service provider limits, garbage collection of unused repositories and automated handling of this error case to mitigate against complete halts in our workloads.

We apologize for the disruption to your builds and understand the impact of our service disruptions.