Introduction

bedevil (bdvl), according to the GitHub page, is an LD_PRELOAD rootkit. Therefore, this rootkit runs in userland. The group Muddled Libra used bedevil to target VMware vCenter servers, according to Palo Alto’s Unit42 Blog, 2024. The rootkit comes with a nifty feature called Dynamic Linker Patching:

Upon installation, the rootkit will patch the dynamic linker libraries. Before anything, the rootkit will search for a valid ld.so on the system to patch. When running ./bdv uninstall from a backdoor shell, the rootkit will revert the libraries back to having the original path. (/etc/ld.so.preload)

This topic is interesting from both a red- and blue-team perspective, primarily because the LD_PRELOAD technique has been well-known for many years. Most detection tools and scripts are typically capable of identifying a shared library path in the ld.so.preload file or detecting a non-empty LD_PRELOAD environment variable. However, patching the dynamic linker provides attackers with significant advantages for stealth, while simultaneously creating new challenges for defenders attempting to identify such intrusions.

In this blog post, we will conduct an in-depth analysis of the patching technique used by the bedevil rootkit, exploring how it works and the advantages that dynamic linker patching offers to attackers.

Patching the dynamic linker

The dynamic linker (also known as the dynamic loader) is a key component of Unix-like operating systems, including Linux, responsible for loading and linking shared libraries to programs at runtime. It allows applications to use code from shared libraries (like libc.so, which contains standard C library functions) without needing to embed this code directly into the executable.

In the setup.py file of the bedevil (bdvl) rootkit repository, there is a configuration flag that determines whether the dynamic linker should be overwritten during the installation process. By default, this flag is set to True, meaning that the dynamic linker will be replaced.

# patch the dynamic linker libraries as to overwrite 
# the original /etc/ld.so.preload path with our own.
# setting to False will instruct the rootkit to use 
# /etc/ld.so.preload instead.
PATCH_DYNAMIC_LINKER = True

In the header file ldpatch.h from the bedevil (bdvl) rootkit repository, two functions are defined. The first function, _ldpatch, is responsible for modifying or patching the dynamic linker (loader), effectively altering its behavior to facilitate the rootkit’s operations. The second function, ldpatch, is designed to reverse these modifications, restoring the dynamic linker to its original state when a login with the backdoor functionality occurs.

Additionally, the ldhomes array lists standard library paths where dynamic linkers can be found. The ldfind function searches these paths to locate linkers.

inc/util/install/ldpatch/ldpatch.h (Source)

static char *const ldhomes[7] = {"/lib", "/lib32", "/lib64", "/libx32",
                                 "/lib/x86_64-linux-gnu", "/lib/i386-linux-gnu", "/lib/arm-linux-gnueabihf"};

char **ldfind(int *allf);
#include "find.c"

int _ldpatch(const char *path, const char *oldpreload, const char *newpreload);
int ldpatch(const char *oldpreload, const char *newpreload);
#include "patch.c"

We will analyse key components of the file patch.c. next.

inc/util/install/ldpatch/patch.c (Source)

The function _ldpatch takes three parameters:

• path: The path to the file which gets patched, for example, /lib64/ld-linux-x86-64.so.2
• oldpreload: The old LD_PRELOAD value inside the file
• newpreload: The new LD_PRELOAD value, which replaces the old value

int _ldpatch(const char *path, const char *oldpreload, const char *newpreload){

It checks next if the length of oldpreload matches the length of newpreload. If the lengths don’t match, the function returns 0 and exits early. Because the patching occurs in place, altering the file size would complicate the process or even break the process altogether.

if(strlen(oldpreload) != strlen(newpreload))
    return 0;

Here is the main code that overwrites the dynamic linker file(s), part of patch.c:

int count = 0,   // Counter to track matching characters in oldpreload
c = 0;       // Counter to track position in newpreload string

do{
    n = fread(buf, 1, fsize, ofp);  // Read from the original file into buf
    if(n){
        for(int i = 0; i <= fsize; i++){
            if(buf[i] == oldpreload[count]){
                if(count == LEN_OLD_PRELOAD){  // If we match the entire oldpreload string
                    for(int x = i-LEN_OLD_PRELOAD; x < i; x++)
                        memcpy(&buf[x], &newpreload[c++], 1);  // Overwrite with newpreload
                        break;  // Stop after the replacement
                    }
                    count++;
                } else count = 0;  // Reset if there's a mismatch
            }
        }else m = 0;
}while(n > 0 && n == m);  // Loop until all file content is processed

The value of oldprelaod is defined in the setup.py file (/etc/ld.so.preload)). Inside setup.py, we find the line: 'PRELOAD_FILE':ut.randpath(18). Thus, the path of the new LD_PRELOAD file is not static and will be different on every machine, therefore hindering threat hunting for this particular value. In essence, the code scans for various dynamic loaders on the compromised system, loads the files into memory, and searches for occurrences of the string /etc/ld.so.preload. It then replaces this string with a new, randomly generated path. This path will hold the new path to a shared library, which is responsible for hijacking various system calls.

Investigation

Now, to the most important question :) How do we find out if somebody tampered with our dynamic loaders when investigating a potentially compromised server? Tony Lambert on X has one possible answer to that question (original tweet here):

Quote: If you get a match, your linker probably hasn’t been tampered with, if no match you might want to check its strings to see if it has a seemingly random file within. Let’s explore a different technique.

strace

The dynamic linker processes the shared library specified with the LD_PRELOAD environment variable or the content of the specified ld.so.preload file before any other library or function is invoked. This is why LD_PRELOAD libraries are always loaded first. See the following strace output, where we look at the invoked system calls of ls.

# strace ls
execve("/bin/ls", ["ls"], 0x7ffeb123ea50 /* 20 vars */) = 0
brk(NULL)                               = 0x20b6000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbfa6f79000
access("/bin/filebeathmili", R_OK)      = 0
open("/bin/filebeathmili", O_RDONLY|O_CLOEXEC) = 3

Have you spotted something anomalous? We found the malicious LD_PRELOAD file! The first access syscall tries to load the file specified as the new LD_PRELOAD file, which is the path the rootkit overwrote in the dynamic loader. As this path was randomly generated during the initialization of the rootkit, we had no prior knowledge of this path.

On success (all requested permissions granted, or mode is F_OK and the file exists), zero is returned (access(2) man page). This is proof that this file is present on disk and readable.

Here, as a comparison, the strace output of an untampered system:

# strace ls
execve("/usr/bin/ls", ["ls"], 0x7fff03446860 /* 23 vars */) = 0
brk(NULL)                               = 0x58d4ba5d5000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7b35ec707000
access("/etc/ld.so.preload", R_OK)      = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3

However..

[root@dfir]# xxd /bin/filebeathmili
xxd: /bin/filebeathmili: No such file or directory
[root@dfir]# cat /bin/filebeathmili
cat: /bin/filebeathmili: No such file or directory

Bummer. 🤔 The rootkit is doing what it’s tasked to do. Protect the content of the malicious LD_PRELOAD file.

Static binaries

In a statically linked binary, LD_PRELOAD has no effect because all the necessary code is already compiled directly into the program. Since the binary does not search for external libraries, a rootkit cannot inject its code using this method. In our lab machine, we obtained some Linux static binaries from GitHub.

find

Previously, cat and xxd returned “No such file or directory” because the rootkit had hooked various system calls, causing errors whenever we attempted to examine the contents of the preload file. However, we have now discovered our hidden file within the /bin directory.

[root@dfir bin]# ./find.1 . | grep filebeathmili
./filebeathmili

Using a statically compiled strings executable, we were able to extract the path to the malicious .so file:

strings

[root@dfir bin]# ./strings.1 filebeathmili
/bin/nl-monitp/libnl-monitp.so.$PLATFORM

Once again, using a dynamically linked tool (cp) to copy the malicious shared object file to a temporary location proved unsuccessful. However, by using a statically linked tool, we were able to copy the file to a different location for further examination.

[root@dfir]# ./cp.1 /bin/nl-monitp/libnl-monitp.so.x86_64  /tmp/
[root@dfir]# cd /tmp/
[root@dfir]# file libnl-monitp.so.x86_64 
libnl-monitp.so.x86_64: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

ldd

On most Linux systems, we can find the 64-bit dynamic linker at /lib64/ld-linux-x86-64.so.2. This is no exception in our case, as the output of ldd shows

[root@dfir]# ldd /usr/bin/ls
linux-vdso.so.1 =>  (0x00007ffe39b4e000)
[..]
/lib64/ld-linux-x86-64.so.2 (0x00007f3f79c87000)
[..]

Now that we have identified the target file (libnl-monitp.so.x86_64), locating its traces within the dynamic loaders has become much easier.

[root@dfir]# grep filebeathmili *.so*
Binary file ld-2.17.so matches
Binary file ld-linux-x86-64.so.2 matches
Binary file ld-lsb-x86-64.so.3 matches
[root@dfir]# strings ld-linux-x86-64.so.2  | grep filebeathmili
/bin/filebeathmili

Automatic checks

The command rpm -V glibc is used to verify the integrity of the installed glibc package on an RPM-based Linux system (like RHEL, CentOS, or Fedora). The -V (or –verify) option tells RPM to check the package’s files against the metadata in the RPM database. if there are no changes, there will be no output.

The output of the command after the infection:

# rpm -V glibc
..5....T.    /lib64/ld-2.17.so

Each dot (.) means that attribute matches, while a letter or symbol indicates a difference. Here’s what the relevant character means:

• 5: MD5 checksum differs
• T: Modification time differs

When we double-check with the output of strace like before, we see that indeed the first access call reaches out to a malicous looking file:

# strace ls
execve("/bin/ls", ["ls"], 0x7ffd9e1ff160 /* 20 vars */) = 0
brk(NULL)                               = 0xa52000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1cace7c000
access("/bin/prtstatjzbggp", R_OK)      = 0

Since we installed the rootkit manually, we were able to observe the installation paths during the process. Unsurprisingly, these paths matched exactly what we saw in the output, confirming that this could be an effective detection method for this specific type of rootkit. A similar approach can be applied on Debian-based systems by using debsums to verify the integrity and detect any unauthorized modifications.

Honorable mentions

A newer rootkit, OrBit, first detected by Intezer (see New Undetected Linux Threat Uses Unique Hijack of Execution Flow), also have some possibiliites to modify the dynmaic loader. Here is a detailed analysis of this rootkit.