Containers make it very easy to package and deliver applications, so it’s not surprising that many ISVs (Independent Software Providers) are leading the trend of packaging their software, whether it’s commercial off-the-shelf (COTS) or custom-developed code, as container images. These images are then fed into the enterprise customer’s CI/CD pipeline, and are often treated in the same way as home-grown images. In fact, at Aqua, we have several enterprise customers who work closely either as consumers of these 3rd party images, or as their suppliers, for example, in the banking and Fintech sectors.
While many ISVs are looking for new ways to test product security before delivery to customers, when it comes to third-party apps in general (and I’m not even referring to containers yet), organizations usually overlook the security of these apps, relying on their provider to run the required security checks.
Additionally, regulatory mandates like PCI DSS and others, require implementation of a third-party software security assurance program to ensure that data and apps are secure and compliant.
If we move this reality into the container environment, the fact that a single organization may have numerous third-party apps running in its environment in a microservices architecture can translate into thousands of third-party containers. Your runtime environments will include (if they don’t already) a multitude of containers that are instantiated from third-party images.
Tips for Securing ISV-Provided Images and Delivering Safe Images to Customers
1. Scan every image
As third-party images aren’t part of your build process but added to the registry post-build, you’d enable DevOps practitioners to pull images from third party registries and scan them directly from Jenkins or any other CI tool. You might also find it useful to add a scanner CLI tool as an image to the relevant registry/ies. This will allow for on-demand scanning of new images which are added outside of the pipeline. Your container security solution should be configured to allow/disallow images to be used based on your security policy (e.g. block images with blacklisted OSS license).
2. Apply “good old security best practices”
It’s not just about known vulnerabilities, there are other risks you should eliminate as much as possible from your images:
- Ensure images do not have embedded secrets
- Prevent configuring images from running with excess privileges (e.g. configured as a ‘root’ user)
- Block images with blacklisted vulnerabilities
- Block images by CVSS score, or those with a high severity
- Ensure that images do not include, for example, HIPAA/PCI-sensitive data
3. Use labelling
Uniquely label your third-party images so you can easily enforce and apply different security and access permissions policies based on these labels. In addition, by labeling your images and the containers instantiated from them, you can then configure your orchestrators to separate containers and hosting zones, and deploy them on separate network segments or Kubernetes namespaces. This is how you prevent the mix of workloads and segregate them according to sensitivity level, purpose, and threat posture.
4. Verify integrity and provenance
Ensure that an ISV-provided image hasn’t been intercepted or modified before you deploy it.
Tools like Grafeas and Notary can help you verify the provenance of an image. Grafeas is an open source API for metadata, which can be used to store “attestations” or image signatures, which can then be checked as part of admission control. Notary is an implementation of The Update Framework that ensures you pull the signed, correct version of an image.
5. Apply runtime protection to third-party containers.
You must be able to know a container’s expected behavior and lock it down to effectively detect anomalies. This can be done automatically by creating a security profile based on container activity. It consists of file access, resource usage, read-only root filesystem, network settings, namespace settings, seccomp profile, and executables, baselining all legitimate container activity and using machine learning to create a whitelisting policy. This creates a least privilege security runtime profile so that even if there’s a suspicious inbound/outbound container communication or a file is added to the container, you can immediately detect and block it with no container downtime to support app business continuity.
Many Upsides, No Excuses
By automating these functions/best practices and embedding them into your pipeline and runtime environments, you’re not only maintaining pipeline hygiene and minimizing the attack surface, but you’re also able to enforce consistent security policies across the board, ensuring every image used in your environment is scanned and authorized based on your security policies, whether it’s a third-party or created in-house image.
If you’re consuming 3rd-party images, it’s your responsibility to make sure they are up to scratch security-wise. Likewise, if you’re supplying the images, but the policy requirement must come from the end-user of those images. With free tools like Aqua’s MicroScanner, there are really no excuses not to take these basic steps to reduce the risks of using vulnerable images.
