Last night, we were made aware that environment variables configured on CodeShip Basic builds with pre-configured deployment pipeline options (such as Heroku, AWS and Google App Engine) were being printed in your build logs. This was an unintended side effect of recent infrastructure work.
Immediately after being notified, our on-call team determined that the issue began several weeks ago as part of a new backend system rollout for handling build logs. Within minutes, we turned off this new log function as a short term fix and deployed a permanent solution later in the evening.
There are several important things to note about what was, and wasn’t exposed so that you can make the appropriate decision on what keys you may need to rotate.
- This problem impacted CodeShip Basic builds using one of our pre-configured deployment pipelines. CodeShip Pro and Basic custom script deployments were not impacted.
- No external user had access to data at any point, as our logs are not available via API and are not downloadable. Only team members you have added to your projects on CodeShip had access to the build logs which included the secrets - but these logs were accessible by all team members.
- We have scrubbed all the relevant build logs, including those from all our backup systems, for the entire impacted period of time.
The root cause of the issue was that our new logging system was not accounting for the deployment context of these specific types of CodeShip Basic steps, which is an oversight of both automated and manual testing on our end.
We are working to add more precise testing around all relevant scenarios, and as our longer-term plans for improving secret handling become clear, we will share them with you for feedback. We’re sorry for the issue - we take security very seriously and our team has been working as hard as possible to resolve the issue and put steps in place to prevent any possible recurrence in the future.
Posted about 1 month ago. Feb 21, 2019 - 19:06 UTC