A new vulnerability (CVE-2019-11246) was disclosed that enables path traversal in kubectl, the popular command line interface for running commands on Kubernetes clusters. What’s interesting about this CVE is that we’ve already seen two previous variations of the same vulnerability disclosed and patched. Read on to learn how that happened.
Kubectl is a simple tool that can be used to manage complex Kubernetes clusters, and is therefore used by almost every Kubernetes admin out there. But sometimes simplicity comes at a price and in this case, it seems, a repetitive one. In this post I’ll talk about how kubectl was found vulnerable to attack, in the same way, three times over the past year.
I’ll show how sometimes even the best security researchers and developers can accidentally overlook a vulnerability in the code.
And most importantly, I’ll demonstrate the steps you can take to mitigate this vulnerability.
The Story Begins…
About a year ago, when security researcher Michael Hanselmann discovered a vulnerability in the kubectl cp and oc cp (The equivalent of kubectl in OpenShift) commands, which let you copy files from a pod to your client machine.
The copy mechanism works as follows:
- Kubernetes packs the files inside the container (using the tar binary inside the container)
- Kubernetes copies this tar and sends it to the client
- Kubectl unpacks this tar in the client’s machine
In his research, Michael found that if an attacker could replace this tar binary inside the container (by infecting a base image, or in some other malicious way), he could output a specially crafted tar file that would include files with relative path names, thus performing a Path Traversal on the client machine. By doing that, the attacker could replace files on the machine and potentially even execute code.
Depending on the permissions of the kubectl client, the attacker could gain access to specific resources. In cases where kubectl runs as root, the result could be devastating.
This illustration demonstrates the flow:
This vulnerability was assigned CVE-2018-1002100 with a high risk severity rating.
Kubernetes and OpenShift rushed to fix this bug, and published appropriate fix versions. Essentially, the fix contained a call to cp.go:clean from the untarAll function that strips the path traversal characters.
Fast forward one year later…
Our story continues as a new vulnerability was discovered a few months ago. When patching the first bug, the developers unknowingly left another security hole.
The new vulnerability exposed the same the Path Traversal risk, but now using a malformed tar file that contains symlinks (Symbolic Links) to paths in the client machine. Thus, potentially leading to the damage done in the original bug. The blue highlights represent the differences between the original vulnerability and the new one.
The new vulnerability was assigned CVE-2019-1002101. It was fixed in the cp.go:untarAll function by simply verifying symlink paths, where they existed. Kubernetes released multiple fix versions for this, so all we had to do is update our kubectl and be safe. right?
The Present
A couple of days ago, Charles Holmes from Atredis Partners discovered that the developers did not actually patch the symlink vulnerability either.
This discovery resulted in a new CVE-2019-11246 being issued, and new fix versions for the kubectl client.
We don’t have much official information about this issue, as the announcement stated only that: “the details for this vulnerability are very similar to CVE-2019-1002101. The original fix for that issue was incomplete and a new exploit method was discovered.”
While it is unclear how that new exploit works and what are the specific “unpatched” holes were, by looking at the fix we can see that much more work was done this time, and a lot of the relevant code in the cp.go:untarAll function was rewritten as well.
Let’s get you checked
You can check if you have a vulnerable version with kube-hunter, our open source penetration testing tool. We’ve added a new hunter that checks your kubectl version for these two vulnerabilities.
All you need to do is:
- Get the latest update from kube-hunter’s Github repo
- Run kube-hunter on your client host machine (make sure kubectl is in your PATH)
It doesn’t matter which mode you run it with, any run of kube-hunter will search for these vulnerabilities.
If you see those vulnerabilities in the report, you need to upgrade your client to the latest version. Kube-hunter will output your current kubectl version, in the ‘evidence’ part of the report.
How to Mitigate
If kube-hunter found you vulnerable, you will need to update your kubectl client to one of the following versions: 1.12.9, 1.13.6, 1.14.2
Additional mitigation with Aqua CSP
- To protect against replacing the tar binary with a malicious copy, we can use Aqua’s Drift Prevention feature that will block any modified/new binary from running inside the container, thus mitigating the exploitation of this vulnerability.
- Use Aqua’s Image Assurance policies to block untrusted images from running. Since this attack could originate from a malicious image, blocking untrusted images would mitigate most cases.
By using one of those options, you can make sure that even clients that are running a vulnerable version of kubectl on your clusters will not be affected.
To Summarize
The Kubernetes community is growing, and we can see that security awareness is growing with it. Although the vulnerability was not fully patched last year, with growing awareness and relentless research it is now properly patched thanks to multiple contributions from the community.
