Identified - Dear CodeShip users,
We are reaching out to inform you of additional information we have uncovered as a result of our continuing investigation of the recent GitHub breach. To provide maximum transparency, we are reporting on the results of our investigation, the impact on users, actions you must take to protect yourself/your organization, and actions we will take to strengthen our security processes going forward.
On Wednesday, September 16, 2020, CloudBees was notified by GitHub of suspicious activities targeting CodeShip business accounts connected to GitHub via the CodeShip GitHub app and now deprecated CodeShip OAuth tokens. CloudBees immediately initiated an investigation conducted by our security and engineering teams, and on September 27, we identified additional evidence of malicious activity against a failover CodeShip database. On September 29, we uncovered evidence to indicate that a malicious actor had access to this failover instance during the period of June 2019 to June 2020. At this time and to the best of our knowledge, we have no evidence of malicious activity or attempts within CodeShip systems since June 2020.
*What type of data was affected?*
The impacted accounts are those of CodeShip users. No other products or accounts were affected and CodeShip is in no way integrated with other CloudBees products or systems.
*For all CodeShip users:*
CodeShip users hashed account passwords, one-time password (OTP) recovery codes, and the OTP secret keys used to seed two-factor authentication all may have been exposed.
Business contact information for invoicing purposes such as company contact name, company name, VAT number, postal address, phone number also may have been exposed. No payment information, such as bank account numbers or credit card numbers, was exposed. No other CloudBees product other than CodeShip was impacted. Also, the logging system was not accessed for any customers.
*For CodeShip Basic users:*
Any information contained in CodeShip users’ pipelines may have been exposed. This includes scripts, environment variables, access tokens and other similar data.
*For CodeShip Pro users:*
AES encryption keys may have been exposed.
*Steps you should take*
Although at this time we have no evidence that the data potentially exfiltrated has been used, all CodeShip users may have been affected (including free, Basic and Pro accounts) and should take the following steps:
- Immediately rotate any keys or other secrets for cloud providers, third-party tools, or anything else that you used in your CodeShip pipelines.
- If using CodeShip Pro, rotate your AES key and re-encrypt your secrets.
- Immediately identify any other sensitive information that is stored in your pipelines and replace them within your pipelines and on any external systems.
- Determine whether any of your systems accessible from CodeShip have experienced unauthorized access, by contacting your provider or carefully review your access records.
- Verify that the source code held in repositories that are linked to your CodeShip account have retained their full integrity.
- Reset your CodeShip 2FA:
https://documentation.codeship.com/general/about/2fa/At this time and to the best of our knowledge, we have no evidence of malicious activity or attempts within CodeShip systems since June 2020. We are continuing to monitor the situation.
*Steps we are taking*
As soon as we were notified by GitHub on September 16, we proceeded to rotate all our applications' internal secrets and rebuilt all our AWS AMIs. We are continuing to scrutinize our AWS security logs to monitor for suspicious activity, such as outbound connections to known malicious IPs. To date, we have not found any such activity.
We want you to be assured that we are taking steps to increase the security strength of the CodeShip product, including but not limited to:
- Validation that our product threat modeling and large-scope security reviews are systematically implemented.
- Validation that the application of production security standards to all operational processes and artifacts is systematically implemented.
- Enhancement of strict restrictions on access to production data and strict segregation of sensitive data.
- Improvement of existing SIRT processes to ensure faster and better forensic investigation.
*Who to contact *
We will update here with any new developments.
If you still have questions, please contact
security@codeship.com.
Last but not least, I’d like to apologize for the impact this is having on you. In the decade that CloudBees has been operating SaaS applications, we have always taken full responsibility for our products and we do so today. Please be assured that we will do everything we can to prevent this from happening again.
- Sacha Labourey, CEO, CloudBees
Sep 30, 19:24 UTC
Investigating - On Wednesday, September 16, 2020, CloudBees was notified by GitHub of suspicious activities targeting certain CodeShip accounts connected to GitHub via the CodeShip GitHub app and now deprecated CodeShip OAuth tokens. If your GitHub credentials are impacted, you already received or will shortly receive a notification from GitHub informing you of this incident.
The activities point to tokens being used to access the “/user/repos” GitHub API endpoint (
https://docs.github.com/en/free-pro-team@latest/rest/reference/repos#list-repositories-for-the-authenticated-user), which is used to list users’ GitHub repositories, including private repositories. It is possible your repositories were cloned, so please contact GitHub support as soon as possible.
Because the suspicious activities involve user tokens, as a first step in response we revoked all GitHub related tokens and SSH keys to keep all accounts protected. You need to reauthenticate CodeShip with GitHub immediately to avoid a service impact.
*Action Required*
- If you use GitHub to sign in to CodeShip, sign out of all CodeShip sessions and sign back in.
- If you have GitHub projects setup on CodeShip, first remove and reinstall the CodeShip GitHub app:
https://documentation.codeship.com/general/projects/builds-are-not-triggered/#github- Next, generate a new SSH key for each CodeShip project. This will enable CodeShip to clone the repository for builds again:
https://documentation.codeship.com/general/projects/project-ssh-key/We are continuing to investigate the underlying issue and will update our blog to provide more information as soon as we better understand any additional implications and potential root causes.
Thank you.
The CloudBees Team
Sep 29, 13:50 UTC