A recent security incident disclosed by Codecov has again placed the spotlight on supply chain attacks. Looking at the details of the incident and how the system operated, it’s clear that organizations should make changes to how they use third party services as part of their Continuous Integration (CI) pipelines. So what can you do to reduce the risk of this occurring?
What happened with Codecov?
From Codecov’s incident report we can get an idea of how this attack occurred. An attacker was able to get access to a credential via a mistake in how Codecov were building Docker images (as an aside this is a good demonstration of the importance of secrets management in container building!). This credential then let them modify Codecov’s bash uploader script which was either used directly by customers or via Codecov’s other uploaders like their Gtihub Action. The attacker used this modified script to steal credentials from the CI environments of customers using it.
The perils of curl|bash
When we look at how the bash uploader script works, we can see that the recommended method of operation is to add the following line of code to your CI scripts:
bash <(curl -s https://codecov.io/bash)
This is an example of what generally gets called curl bashing, where a script is executed in an environment directly from a URL. This approach to running software has some security problems, which boil down to the set of systems you’re trusting when you use it. The security problems of this type of software installation have been known for some time (for example, here’s a talk I did at OWASP Appsec EU 2015 on this topic).
Whenever you install software, you’re trusting that the authors had good security practices and are not malicious. However, when you curl|bash you’re also trusting that the server holding the software and the cloud account likely managing that server are being operated securely and no unauthorized access to that hosting infrastructure has occurred, as appears to have happened in this case.
These additional layers of trust widen the possibility for attackers to modify software and execute supply chain attacks.
Actions to take
There’s a number of things to do here, ranging from immediate actions for users of Codecov, to more strategic actions to reduce the risk of this kind of incident occurring in the future.
The first action to take is to assess your exposure to Codecov. If you used it, you should consider all the information held in the environments that it had access to, to be at risk of compromise. It seems that the main target for the attacker in this case was credentials, so rotating any credential that was exposed to your CI environments, is a requirement.
Additionally, Codecov have released a list of Indicators of Compromise which can be used when reviewing logs for evidence of exploitation.
The second action to take is, look at your CI environments to see where you’re using tools like Codecov and consider how you can mitigate the risks of an attack on those tools.
From a security standpoint, ideally avoiding dynamically downloading third party scripts during the CI process is a good idea. Having versions of any scripts in your control that you run and update allows for more control, but of course it also adds an overhead of manually updating them.
Where you do need to include scripts dynamically, one option which can help is checking the integrity of any scripts before you use them. Ideally the tools you’re using will have digitally signed releases. If that’s the case, implementing checks of those signatures into your pipelines can be an effective mitigation for unauthorised modifications.
Even if signed releases aren’t available, it’s possible to validate checksums for downloaded files. Codecov themselves recently modified their Github Action, which uses the bash uploader, to validate the checksum of the running script against published values and fail if they don’t match.
Another more strategic recommendation is to review the third party services you’re using as part of your CI and consider their security as part of your evaluation process. Projects like the OSSF Scorecard can be useful tools to quickly get a sense of how the open source projects you’re making use of are doing at adopting common good practices.
Conclusion
Codecov is another in an expanding line of supply chain attacks. Whilst it’s difficult to say which service will be impacted next, it’s clear that this is an ongoing theme in how attackers will seek to gain access to organizations systems and data.
As a result, there is an increasing need to embed supply chain security practices in all areas of your development process, allowing for clear visibility of potential risks, and adding mitigations to processes to reduce these risks where possible.
Learn how Aqua DTA helps protect against supply chain attacks
