Aqua Tracee is an open source runtime security and forensics tool for Linux. It can help you detect suspicious behavior at runtime using an extensive data collection engine and a sophisticated rules engine. You can further read about the development of Tracee in our blog The Story of Tracee: The Path to Runtime Security Tool.
Tracee ships with a handful of valuable signatures for detecting suspicious behavior, and users can easily add their own rules to checks. Recently, we’ve added and revised many signatures to the built-in signature library Tracee ships with. These signatures cover different scenarios and levels of granularity with the goal of helping the community protect workloads at runtime. You can find the full list of signatures in AVD categorized under MITRE ATT&CK techniques.
With these new and revised signatures you can run Tracee as an out-of-the-box security tool to detect threat actors’ activity in your environments. As an attacker, you would execute your attack with different techniques and tools that allow you to maximize the impact. Many of these can be generally classified into a commonly covered kill-chain such as the MITRE ATT&CK framework. Let’s explore a typical attack through the ATT&CK framework while considering how Tracee, with its revised signature library, can address it.
Initial Access
The first stage of the attack is to gain initial access to the target environment. In the cloud you mainly gain access via vulnerable software, a misconfigured environment, or through a supply chain. In these instances, attackers often try to exploit public-facing applications. For that, Tracee offers the signature ‘web server spawned a shell’. Whether the attacker exploited a vulnerability, misconfiguration, or a supply chain, if your web server shows the behavior of spawning a shell, Tracee would detect it.
Privilege Escalation
In the cloud, once threat actors gain initial access they often seek to escalate privileges. Containers are ephemeral by nature and jailed to their specific namespace and cgroup. Thus, attackers are looking to escape to the host to gain further (possibly full) control over the system. Since we understand how attackers think, in this release we added five new signatures that are looking specifically for that. These are the signatures that were designed to detect escape to host attempts:
- Core Dumps Configuration File Modification Detected
- Cgroups Release Agent File Modification
- Kcore Memory File Read
- System Request Key Configuration Modification
- Container Device Mount Detected
These are real life scenarios that we often see when analyzing attacks in the wild. We have written about our findings in our blog Threat Actors Using release_agent Container Escape.
In addition, Tracee signatures also contain detection of sudoers file modification that may lead to higher privileges, ASLR file inspection, and Docker socket abuse – which can lead to privilege escalation.
Persistence
Once the attackers reach the host, we often see that they are trying to gain persistence, commonly by creating cron jobs, loading kernel modules, or injecting code to LD_preload. (These previous links show how these techniques are used by attackers in the wild). Tracee contains three signatures that look exactly for that.
In many campaigns, we see attackers are trying to open reverse shell, which is when an interactive shell is invoked from a target machine back to the attacker’s machine, giving it interactive control over the target. There are several ways to execute reverse shell opening. One of them is detected with the signature Process Standard Input Output Over Socket Detected.
Defense Evasion
Another prevalent adversarial behavior we see is defense evasion. In this release, we included a dozen signatures aimed at detecting such behavior. One signature is tailored to detect suspicious/malicious behavior in containers. Since containers are meant to be immutable by nature, we shouldn’t expect to see new executables appearing during runtime. Tracee can detect this exact behavior.
In addition, there are more sinister behaviors that may indicate a presence of rootkits in the system. Following vast research of Aqua Nautilus on this topic, we found several behaviors that are indicative of rootkits. We have now released a couple of signatures that detect this behavior: File Operations Hooking On Proc Filesystem Detected and Syscall Table Hooking Detected. On top of that, we also added fileless execution detection, which is also a powerful technique actively used in the wild by attackers to avoid detection by many security tools.
In this release, we boosted Tracee users’ capability to detect attacks on their workloads. We also augmented Tracee’s code base with particular key signatures that allow practitioners to use Tracee as an out-of-the-box security tool in container runtimes. Further information can be found in the release notes. These signatures are based on Aqua’s expert security research. Team Nautilus researchers are constantly monitoring attackers behavior in the wild to gather the latest data about emerging tactics and techniques. Our threat intelligence is continuously translated into new signatures that are uploaded into Aqua runtime protection solutions and appended to the ever-growing library of hundreds of signatures that will protect Aqua’s customers from an evolving set of scenarios.
Below is a list of the new signatures appended to Tracee’s rules:
ASLR
inspection detected – TheASLR
(address space layout randomization) configuration was inspected. ASLR is used by Linux to prevent memory vulnerabilities. An adversary may want to inspect and change theASLR
configuration to avoid detection.Cgroups notify_on_release
file modification – An attempt to modifyCgroup notify_on_release
file was detected. Cgroups are a Linux kernel feature which limits the resource usage of a set of processes. Adversaries may use this feature for container escaping.- Core dumps configuration file modification detected – Modification of the core dump configuration file (
core_patter
n) detected. Core dumps are usually written to disk when a program crashes. Certain modifications enable container escaping through the kernel core_pattern feature. - Default dynamic loader modification detected – The default dynamic loader has been modified. The dynamic loader is an executable file loaded to process memory and run before the executable to load dynamic libraries to the process. An attacker might use this technique to hijack the execution context of each new process and bypass defenses.
- Docker socket abuse detected – An attempt to abuse the Docker UNIX socket inside a container was detected.
docker.sock
is the UNIX socket that Docker uses as the entry point to the Docker API. Adversaries may attempt to abuse this socket to compromise the system. - Hidden executable creation detected – A hidden executable (ELF file) was created on disk. This activity could be legitimate; however, it could indicate that an adversary is trying to avoid detection by hiding their programs.
Kcore
memory file read – An attempt to read /proc/kcore file was detected.KCore
provides a full dump of the physical memory of the system in the core file format. Adversaries may read this file to get all of the host memory and use this information for container escape.- Process memory access detected – Process memory access detected. Adversaries may access other processes memory to steal credentials and secrets.
- Rcd modification detected – The
rcd
files were modified. rcd files are scripts executed on boot and runlevel switch. Those scripts are responsible for service control in runlevel switch. Adversaries may add or modify rcd files in order to persist a reboot, thus maintaining malicious execution on the affected host. sched_debug CPU
file was read – Thesched_debug
file was read. This file contains information about your CPU and processes. Adversaries may read this file to gather that information for their use.- Scheduled tasks modification detected – The task scheduling functionality or files were modified.
Crontab
schedules task execution or enables task execution at boot time. Adversaries may add or modify scheduled tasks in order to persist a reboot, thus maintaining malicious execution on the affected host. - Sudoers file modification detected – The sudoers file was modified. The sudoers file is a configuration file which controls the permissions and options of the sudo feature. Adversaries may alter the sudoers file to elevate privileges or execute commands as other users or spawn processes with higher privileges.
- System request key configuration modification – An attempt to modify and activate the System Request Key configuration file was detected. The system request key allows immediate input to the kernel through simple key combinations. Adversaries may use this feature to immediately shut down or restart a system. With read access to kernel logs, host related information such as listing tasks and CPU registers may be disclosed and could be used for container escape.
Signatures that were modified to improve performance:
- Kubernetes API server connection detected – A connection to the Kubernetes API server was detected. The K8S API server is the brain of your K8S cluster. Adversaries may try and communicate with the K8S API server to gather information/credentials, or even run more containers and laterally expand their grip on your systems.
- Process standard input/output over socket detected – A process has its standard input/output redirected to a socket. This behavior is the base of a Reverse Shell attack, which is when an interactive shell is invoked from a target machine back to the attacker’s machine, giving it interactive control over the target. Adversaries may use a Reverse Shell to retain control over a compromised target while bypassing security measures like network firewalls.
Signatures that were re-written from rego to golang:
- Web server spawned a shell – A web-server program on your server spawned a shell program. Shell is the Linux command-line program. Web servers usually don’t run shell programs, so this alert might indicate an adversary is exploiting a web server program to gain command execution on the server.
- Kernel module loading detected – Loading of a kernel module was detected. Kernel modules are binaries meant to run in the kernel. Adversaries may try and load kernel modules to extend their capabilities and avoid detection by running in the kernel and not user space.
- K8s TLS certificate theft detected – Theft of Kubernetes TLS certificates was detected. TLS certificates are used to establish trust between systems. The Kubernetes certificate is used to enable secure communication between Kubernetes components such as kubelet scheduler controller and API Server. An adversary may steal a Kubernetes certificate on a compromised system to impersonate Kubernetes components within the cluster.
LD_PRELOAD
code injection detected –LD_PRELOAD
usage was detected.LD_PRELOAD
lets you load your library before any other library, allowing you to hook functions in a process. Adversaries may use this technique to change your applications’ behavior or load their own programs.- File operations hooking on proc filesystem detected – File operations hooking on proc filesystem detected. The proc filesystem is an interface for the running processes as files. This allows programs like ps and top to check what the running processes are. File operations are the functions defined on a file or directory. File operations hooking includes replacing the default function used to perform a basic task on files and directories like enumerating files. By hooking the file operations of /proc, an adversary gains control of certain system functions such as file listing or other basic functions performed by the operation system. The adversary may also hijack the execution flow and execute its own code. File operation hooking is considered a malicious behavior that is performed by rootkits and may indicate that the host’s kernel has been compromised. Hidden modules are marked as hidden symbol owners and indicate further malicious activity of an adversary.
- Code injection detected through
/proc/<pid>/mem
file – Possible code injection into another process was detected. Code injection is an exploitation technique used to run malicious code; adversaries may use it in order to execute their malware. - Code injection detected using
process_vm_writev
syscall – Possible code injection into another process was detected. Code injection is an exploitation technique used to run malicious code; adversaries may use this in order to execute their malware. - Code injection detected using ptrace – Possible code injection into another process was detected. Code injection is an exploitation technique used to run malicious code; adversaries may use it in order to execute their malware.
- Anti-Debugging detected – A process used anti-debugging techniques to block a debugger. Malware uses anti-debugging to stay invisible and inhibit analysis of their behavior.
- Cgroups release agent file modification – An attempt to modify a Cgroup release agent file was detected. Cgroups are a Linux kernel feature which limits the resource usage of a set of processes. Adversaries may use this feature for container escaping.
- Container device mount detected – Container device filesystem mount detected. A mount of a host device filesystem can be exploited by adversaries to perform container escape.
- New executable dropped – An Executable file was dropped in the system during runtime. Container images are usually built with all binaries needed inside. A dropped binary may indicate that an adversary infiltrated your container.
- Dynamic code loading detected – Possible dynamic code loading was detected as the binary’s memory is both writable and executable. Writing to an executable allocated memory region could be a technique used by adversaries to run code undetected and without dropping executables.
- Fileless execution detected – Fileless execution was detected. Executing a process from memory instead from a file in the filesystem may indicate that an adversary is trying to avoid execution detection.
- Syscall table hooking detected – Syscall table hooking detected. Syscalls (system calls) are the interface between user applications and the kernel. By hooking the syscall table, an adversary gains control of certain system functions, such as file writing and reading or other basic functions performed by the operation system. The adversary may also hijack the execution flow and execute its own code. Syscall table hooking is considered a malicious behavior that is performed by rootkits and may indicate that the host’s kernel has been compromised. Hidden modules are marked as hidden symbol owners and indicate further malicious activity of an adversary.