We are continuing to monitor build behaviour vs. AWS status updates.
* the error rate on AWS backend services is continuing to drop, but has not reached normal levels (typically 0 in some parts of the system) * new builds are completing in a timely manner, although stalled builds may exist on your account * if your build has stalled, please cancel (and restart it if you need that build to complete)
Posted Nov 26, 2020 - 06:19 UTC
Update
AWS is now declaring their primary outage of Kinesis to be over - however they will take time to catch up. We use Kinesis for high availability (ordinarily!) of our build log systems. We are also dependent on CloudWatch to detect some service issues (and AWS has not declared that outage to be over yet).
Builds are processing, but we are seeing long delays in pushing logs through the system - sometimes causing timeouts etc. We expect the behavior to improve over the next few hours and we will continue to monitor.
Thank you for your patience.
Posted Nov 26, 2020 - 03:51 UTC
Monitoring
There is an ongoing outage with multiple AWS services - please see https://status.aws.amazon.com. AWS is reporting that service restoration is underway and services are returning to normal operation (although throttled).
This concurs with our internal service monitoring showing services returning to normal operation.
You may experience some delay in build notifications and log access while services continue to be restored by AWS.
Posted Nov 26, 2020 - 01:33 UTC
Identified
There is an ongoing outage with multiple AWS services - please see https://status.aws.amazon.com. AWS is currently reporting it may be several more hours until those services are fully restored.
These system outages are directly impacting CodeShip customer builds (execution / monitoring / scaling).
We are seeing incremental improvements in build processing as AWS restores services (even though they are not always updating their status) - we will update once we know more.
Posted Nov 26, 2020 - 00:16 UTC
Update
We are closely monitoring the various service outages reported by AWS: https://status.aws.amazon.com. The ECS and Kinesis outages are directly impacting CodeShip customer build logs and preventing builds from running. AWS is currently reporting it may be several more hours until those services are fully restored.
Posted Nov 25, 2020 - 20:15 UTC
Update
We are continuing to investigate this issue.
Posted Nov 25, 2020 - 18:51 UTC
Investigating
We are currently investigating this issue.
Posted Nov 25, 2020 - 15:51 UTC
This incident affected: CodeShip Pro (Builds) and CodeShip Basic (Builds).