For more than two months, an unidentified attacker was able to access and modify the Codecov Bash Uploader script, a tool that many organizations and open-source projects use as part of their testing processes. As a result, the attacker may have had the ability to find and export sensitive information from Codecov customers’ continuous integration environments, including keys, tokens, and secrets.
Codecov, which makes code coverage checking tools, said that a customer informed the company of the issue on April 1 after noticing a discrepancy between the SHA1 sum for the uploader posted publicly on GitHub and the sum calculated from the Bash Uploader they had downloaded. Codecov began investigating the incident and disclosed it on Thursday.
“The actor gained access because of an error in Codecov’s Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script,” Jerrod Engelberg, CEO of the company said in a statement.
“Our investigation has determined that beginning January 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users' continuous integration (CI) environments. This information was then sent to a third-party server outside of Codecov’s infrastructure.”
The attacker could have had access to a significant range of sensitive data from customers' environments, including credentials, tokens, or keys, and any of the services, apps or datastores that are accessed with those credentials. As a result, the company recommends that affected customers rotate all of the keys, credentials, and tokens that were in affected environmental variables in their CI processes.
“Our investigation has determined that beginning January 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script."
Engelberg said the company sent email notifications Thursday to all of the customers that the company has been able to confirm were affected by the breach. But the intrusion could have broad downstream effects too, given that the Codecov Bash Uploader is used by numerous open-source projects. On its site, Codecov lists several companies that produce popular open source projects as customers, including Mozilla, Elastic, and SaltStack. Another project that uses Codecov is Python, but the maintainers said Thursday that the project was not affected.
“We use codecov across many many testing jobs. However, none of these jobs contain access to any secrets or tokens or any sort. Further, we do not not use codecov in any jobs that generate release artifacts (e.g. built wheels). Because our CI infrastructure relies on ephemeral environments; jobs are isolated from each other -- gaining access to a job that runs tests cannot be pivoted to access to a job that generates a release wheel. 100% of our source code is Open source, including all release infrastructure, so there was no source code to steal,” Alex Gaynor said in an email to a Python mailing list.
Engelberg said Codecov is working with law enforcement as well as an outside forensics firm to determine the extent ot the intrusion.