Overview
This note summarizes manual steps and checks to determine whether a Linux system has been compromised. The approach relies on audit logs, threat intelligence, file integrity checks, command auditing, and analysis of system and application logs.
Audit and log sources
- Audit logs (system call monitoring, process creation, hidden processes).
- Network access logs and threat intelligence (suspicious IPs, files, hidden files, hidden processes, SUID files).
- File integrity checks and command auditing (evaluate operations via security libraries).
- Malware databases (backdoors, rootkits, viruses, trojans).
- Version control checks: verify whether GitLab code was modified using git diff and code commit logs.
- Server logs: check for hijacking, login records, and unauthorized SQL execution.
- /var/log/secure: check for successful remote logins from unexpected IPs.
Initial file and process checks
- Use md5 checksums to detect modified files.
- Review logs for deletions or unusual entries; inspect audit records for abnormal syscalls.
- Inspect suspicious processes and listening ports; check bastion host logs for abnormal operations such as file transfers or unauthorized installations.
- Use a network intrusion detection system (NIDS) to find anomalous network behavior.
- Run rpm -Va to see which packages have been replaced or altered.
Network and kernel inspection
- Identify outbound IPs for the machine. Capture traffic with tcpdump and analyze with Wireshark.
- Use threat intelligence to investigate outbound IPs.
- Inspect loaded kernel modules.
- For executables, check linked libraries using ldd and trace system calls with strace.
- Inspect DNS logs for malicious domain lookups and application logs for indicators of compromise.
- Find files modified within suspected compromise time windows and analyze them.
Login and user account checks
- Inspect /etc/passwd and user information. Use who, w, uptime to see logged-in users. Check for privilege users (uid=0).
- Scan for weak passwords on SSH and all applications; perform baseline vulnerability scans for major issues such as unauthorized access.
- Inspect sudoers configuration: more /etc/sudoers.
- Search for all SUID and GUID files: find . -perm /2000 and find . -perm /4000.
- Check SSH authorized keys: /root/.ssh/authorized_keys and known hosts: /root/.ssh/known_hosts to see which keys allow passwordless login.
- Inspect sshd binary and process (ls -al /usr/sbin/sshd; cat /usr/sbin/sshd) and check for replaced system commands or libraries.
- Check /etc/nologin to ensure it has not been replaced by bash, which would allow all users to log in.
Startup and persistence locations
- Inspect startup files and scripts: /etc/profile, .bash_profile, .bashrc, /etc/rc.d/rc.local, /etc/rc.d/init.d, /etc/inittab, /etc/rc.d/rc.sysinit, /etc/rc.d/rc*.d, /etc/init.d, /etc/fstab, /etc/bashrc, ~/.bash_login, ~/.profile, ~/.xinitrc, ~/.xserverrc.
- List /tmp and /dev/shm for suspicious files; these directories allow world-writable uploads and are common for dropped backdoors.
- Inspect cron jobs: /var/spool/cron/*, /etc/crontab, /etc/cron.d/*, /etc/cron.daily/*, /etc/cron.hourly/*, /etc/cron.monthly/*, /etc/cron.weekly/*, /etc/anacrontab, /var/spool/anacron/*.
- Check /etc/rc.d/ and system init scripts for malicious entries.
Relevant log files
- /var/log/cron: scheduled task logs.
- /var/log/cups: printing logs.
- /var/log/dmesg or dmesg: kernel boot and self-test messages.
- /var/log/maillog: mail logs.
- /var/log/messages: most important system messages; the first file to inspect when troubleshooting.
- /var/log/btmp: failed login attempts (binary; use lastb to view).
- /var/log/lastlog: last login times for all users (binary; use lastlog to view).
- /var/log/wtmp: persistent login/logout and boot events (binary; use last to view).
- /var/log/utmp: current logged-in users (use w, who, users to query).
- /var/log/secure: authentication and authorization events (SSH logins, su, sudo, user additions and password changes).
Malware and rootkit scanning
- Run rootkit checks: chkrootkit, rkhunter, ClamAV, and other available scanners. Example tools mentioned: GSCan.
- Security check scripts: security_check (GitHub), linux security scripts.
- Search for SUID files and check /root/.ssh for unauthorized keys.
File system and executable inspection
- Traverse the entire filesystem and use file to identify ELF files; flag unexpected ELF executables as possible trojans.
- Inspect crontab entries and startup scripts for persistence mechanisms.
- Search for hidden files and suspicious binaries: use unhide to find hidden processes and search for unexpected entries in /etc/profile, /.bash_login, /.bashrc, /etc/bashrc, /etc/profile.d/*.sh.
- Find PHP webshells and check web application directories, especially user-upload directories and avatar directories.
- Use find to locate recently changed files: find / -ctime 1 and other time-based searches.
Network and process monitoring tools
- ps -ef and ps aux to list processes and spot suspicious ones; look for su, bash without a dash, chsh, chfn.
- netstat -anplt to view listening sockets and remote connections.
- lsof to see files used by processes and which process uses a given port.
- iftop to monitor socket-level network traffic; nethogs to monitor per-process network usage.
- strings on binaries to search for embedded IPs or suspicious patterns, e.g. strings /usr/bin/.sshd | egrep '[1-9]{1,3}\.[1-9]{1,3}\.'
- Compare the process list under /proc with ps output to detect hidden processes.
Web and application checks
- Inspect application logs for anomalies and webserver logs for unusual requests.
- Scan PHP files for suspicious constructs: grep for eval($_POST and file_put_contents with POST input.
- Check webserver and application data directories for malicious scripts, especially in upload locations.
Forensic and offline analysis
- For thorough analysis, take the machine offline and run antivirus and rootkit scans (ClamAV, rkhunter, chkrootkit) for tracing and malware identification.
- Collect evidence and perform file integrity comparisons using checksums and package verification tools.
Specific commands and locations to check
# Examples of useful checks and file locations rpm -Va file <path> ldd <executable> strace <pid or command> tcpdump -i <if> -w capture.pcap ps -ef netstat -anplt lsof -i iftop, nethogs find / -perm /2000 find / -perm /4000 find / -ctime 1 cat /etc/ld.so.preload echo $LD_PRELOAD cat /etc/ld.so.cache cat /etc/passwd cat /etc/shadow last, lastlog, lastb, who, w strings <binary> | egrep '<pattern>' more /etc/sudoers ls -al /usr/sbin/sshd vim /var/spool/cron/<user>
Cleanup and hardening recommendations
- When removing malware, execute all necessary cleanup steps in one operation because many trojans include self-recovery features.
- Remove unauthorized SSH keys from all users' .ssh directories except those explicitly required for operations such as bastion access.
- Delete suspicious files in /tmp and related temporary directories.
- Ensure no accounts have empty passwords and remove unnecessary privileged tools such as socat if not required.
- If core system binaries or libraries were replaced, consider rebuilding the system from a trusted image and restoring verified configurations and data.
