Java Debug Wire Protocol (JDWP) is a great way to remotely debug applications during development. However, if enabled when shipped to production, hackers can exploit this mistake by running an arbitrary code that allows initial access or privilege escalation in your production environment. Using Aqua’s Dynamic Threat Analysis (DTA) scanner, we at Team Nautilus detected dozens of misconfigured container images in Docker Hub, which belong to large organizations and may expose them to severe risk when running in production. In this blog, I will show how attackers can exploit this misconfiguration and explain the threats to containers and Kubernetes.
What is JDWP?
Developers are meticulously inspecting their code in order to identify and fix bugs. Usually, Java developers debug their applications locally within their Integrated Development Environment (IDE). But sometimes they need to debug the application remotely. To do this, they can use the Java Debug Wire Protocol (JDWP), which is a protocol for communication between a Java Virtual Machine (JVM) tool interface and a
debugger console.
This feature is disabled by default, but when activated, it allows developers to remotely debug the code. If it was enabled during the development phase and mistakenly released into production while it is still enabled, an attacker can exploit this misconfiguration as an initial access or to perform privilege escalation. From within the network or using the exposed debugging port (should it be left exposed), a remote attacker can instruct the JVM to load arbitrary Java classes into memory and leverage these to achieve remote
code execution.
When enabling JDWP in VMs without restricting inbound traffic, a remote attacker may detect and exploit this misconfiguration. This attack vector was fully covered in a blog by RedTeam Security. But is this really an issue when it comes to containers and Kubernetes?
The short answer is yes, the longer one, it is a bit more complicated. In the case with containers, you first need to set up the debugger and set a port to listen to dt_socket, then expose a port when running
a container.
“Don’t expose Java debuggers in production! Check out Team Nautilus’ interesting research about the risks of leaving open Java debuggers in containers and K8s.”
Share on twitterJDWP in container images
Most images with JDWP misconfiguration that we found contained in the entrypoint JVM start command with -agentlib flag, which defines a port that listens to the Java Debugger. For instance, in the screenshot below, you can see that port 8000 is defined as JDWP’s address.
When running this container image without exposing a port on the host, the misconfiguration cannot be exploited, since the debugger listens to localhost. Therefore, you need to run the command with the flag –net=host or with the flag -p 0.0.0.0:host_port:container_port. This makes the misconfiguration detectable to external scanner and exploitable by an attacker. Mind that the container image tag in the first screenshot below is ‘prod’, which may indicate that it’s being used by the owner in production.
Now, the container is running in production and we can simulate an external attacker behavior. First, we can run a scanning tool. As can be seen below, nmap scan showed that port 8000 is open and listening to Java Debug Wire Protocol.
An attacker can try exploiting this misconfiguration with a couple of pre-made Offensive Security Tools (OSTs) that are designed to exploit this misconfiguration.
One of them has a module in the Metasploit framework. We used Kali with generic Metasploit package and loaded java_jdwp_debugger (as can be seen in the screenshot below). Then, we defined the relevant variables, such as victim host IP and port, attacker’s IP and payload. When we ran Metasploit with the default package, it failed creating a session. So, we decided to use a reverse proxy payload in order to complete
the exploit.
As can be seen in the screenshots above and below, this actually worked as we gained shell access with a root privilege to the container.
Now, once we gained initial access, we can continue the attack by trying to detect and collect some sensitive information or for privilege escalation. For instance, running a post exploitation OST script to collect some credentials (punk.py) or trying to escape to the host with BOB, etc. For further information about these techniques, check out our blog about a market-first container image designed to detect and exploit vulnerable K8s clusters.
JDWP in Kubernetes
Similar to a container image, when running in Kubernetes, the pod needs to run under 0.0.0.0:8090, otherwise it will be listed to localhost (127.0.0.1), and an external attacker won’t be able to gain access. In Kubernetes, the port is open to the network and another pod in the cluster can exploit this misconfiguration. In our case, we used the JDWP-shellifier.py in order to exploit this misconfiguration.
Now that we saw that this misconfiguration can be exploited both in containers and Kubernetes, the remaining question is, can this misconfiguration be found in the wild?
The answer is yes! Using Aqua’s Dynamic Threat Analysis (DTA), team Nautilus scanned open registries in Docker Hub and found dozens of container images with these misconfigurations. While some of the images seem to be related to test environments, others are likely used in production environments and K8s clusters. Over a period of one year, team Nautilus detected 114 distinct Docker Hub accounts that had at least one container image with JDWP misconfiguration. In total, we’ve seen 193 distinct container images.
In the chart above, you can see the detection of the new JDWP misconfigurations on a monthly basis. Clearly, there are dozens of container I’m ages with JDWP misconfigurations, and if they are used in production, it may allow remote attackers to gain initial access to sensitive areas.
Conclusion and mitigation
Even if this misconfiguration seems a bit esoteric, it can easily be exploited by adversaries to gain access into your production environment, allowing them to steal data, damage production, or get initial access to carry out more vicious attacks.
So, what should you do?
- Do not use JDWP in production environment. Start the JVM without any agents enabled. Avoid using the
-agentlib, -Xrunjdwp, and-Xdebugarguments in your production images. - Restrict remote debugging to trusted hosts by modifying the network and security policy configurations to grant appropriate permissions only to those trusted hosts.
- Scan your images for JDWP misconfigurations with a dynamic threat analysis tool like Aqua’s DTA. For example, above you can see JDWP misconfiguration detected by DTA.
