Manual Check for System Intrusion

Overview

This note summarizes manual steps and checks to determine whether a Linux system has been compromised. The approach relies on audit logs, threat intelligence, file integrity checks, command auditing, and analysis of system and application logs.

Audit and log sources

Audit logs (system call monitoring, process creation, hidden processes).
Network access logs and threat intelligence (suspicious IPs, files, hidden files, hidden processes, SUID files).
File integrity checks and command auditing (evaluate operations via security libraries).
Malware databases (backdoors, rootkits, viruses, trojans).
Version control checks: verify whether GitLab code was modified using git diff and code commit logs.
Server logs: check for hijacking, login records, and unauthorized SQL execution.
/var/log/secure: check for successful remote logins from unexpected IPs.

Initial file and process checks

Use md5 checksums to detect modified files.
Review logs for deletions or unusual entries; inspect audit records for abnormal syscalls.
Inspect suspicious processes and listening ports; check bastion host logs for abnormal operations such as file transfers or unauthorized installations.
Use a network intrusion detection system (NIDS) to find anomalous network behavior.
Run rpm -Va to see which packages have been replaced or altered.

Network and kernel inspection

Identify outbound IPs for the machine. Capture traffic with tcpdump and analyze with Wireshark.
Use threat intelligence to investigate outbound IPs.
Inspect loaded kernel modules.
For executables, check linked libraries using ldd and trace system calls with strace.
Inspect DNS logs for malicious domain lookups and application logs for indicators of compromise.
Find files modified within suspected compromise time windows and analyze them.

Login and user account checks

Inspect /etc/passwd and user information. Use who, w, uptime to see logged-in users. Check for privilege users (uid=0).
Scan for weak passwords on SSH and all applications; perform baseline vulnerability scans for major issues such as unauthorized access.
Inspect sudoers configuration: more /etc/sudoers.
Search for all SUID and GUID files: find . -perm /2000 and find . -perm /4000.
Check SSH authorized keys: /root/.ssh/authorized_keys and known hosts: /root/.ssh/known_hosts to see which keys allow passwordless login.
Inspect sshd binary and process (ls -al /usr/sbin/sshd; cat /usr/sbin/sshd) and check for replaced system commands or libraries.
Check /etc/nologin to ensure it has not been replaced by bash, which would allow all users to log in.

Startup and persistence locations

Inspect startup files and scripts: /etc/profile, .bash_profile, .bashrc, /etc/rc.d/rc.local, /etc/rc.d/init.d, /etc/inittab, /etc/rc.d/rc.sysinit, /etc/rc.d/rc*.d, /etc/init.d, /etc/fstab, /etc/bashrc, ~/.bash_login, ~/.profile, ~/.xinitrc, ~/.xserverrc.
List /tmp and /dev/shm for suspicious files; these directories allow world-writable uploads and are common for dropped backdoors.
Inspect cron jobs: /var/spool/cron/*, /etc/crontab, /etc/cron.d/*, /etc/cron.daily/*, /etc/cron.hourly/*, /etc/cron.monthly/*, /etc/cron.weekly/*, /etc/anacrontab, /var/spool/anacron/*.
Check /etc/rc.d/ and system init scripts for malicious entries.

Relevant log files

/var/log/cron: scheduled task logs.
/var/log/cups: printing logs.
/var/log/dmesg or dmesg: kernel boot and self-test messages.
/var/log/maillog: mail logs.
/var/log/messages: most important system messages; the first file to inspect when troubleshooting.
/var/log/btmp: failed login attempts (binary; use lastb to view).
/var/log/lastlog: last login times for all users (binary; use lastlog to view).
/var/log/wtmp: persistent login/logout and boot events (binary; use last to view).
/var/log/utmp: current logged-in users (use w, who, users to query).
/var/log/secure: authentication and authorization events (SSH logins, su, sudo, user additions and password changes).

Malware and rootkit scanning

Run rootkit checks: chkrootkit, rkhunter, ClamAV, and other available scanners. Example tools mentioned: GSCan.
Security check scripts: security_check (GitHub), linux security scripts.
Search for SUID files and check /root/.ssh for unauthorized keys.

File system and executable inspection

Traverse the entire filesystem and use file to identify ELF files; flag unexpected ELF executables as possible trojans.
Inspect crontab entries and startup scripts for persistence mechanisms.
Search for hidden files and suspicious binaries: use unhide to find hidden processes and search for unexpected entries in /etc/profile, /.bash_login, /.bashrc, /etc/bashrc, /etc/profile.d/*.sh.
Find PHP webshells and check web application directories, especially user-upload directories and avatar directories.
Use find to locate recently changed files: find / -ctime 1 and other time-based searches.

Network and process monitoring tools

ps -ef and ps aux to list processes and spot suspicious ones; look for su, bash without a dash, chsh, chfn.
netstat -anplt to view listening sockets and remote connections.
lsof to see files used by processes and which process uses a given port.
iftop to monitor socket-level network traffic; nethogs to monitor per-process network usage.
strings on binaries to search for embedded IPs or suspicious patterns, e.g. strings /usr/bin/.sshd | egrep '[1-9]{1,3}\.[1-9]{1,3}\.'
Compare the process list under /proc with ps output to detect hidden processes.

Web and application checks

Inspect application logs for anomalies and webserver logs for unusual requests.
Scan PHP files for suspicious constructs: grep for eval($_POST and file_put_contents with POST input.
Check webserver and application data directories for malicious scripts, especially in upload locations.

Forensic and offline analysis

For thorough analysis, take the machine offline and run antivirus and rootkit scans (ClamAV, rkhunter, chkrootkit) for tracing and malware identification.
Collect evidence and perform file integrity comparisons using checksums and package verification tools.

Specific commands and locations to check

# Examples of useful checks and file locations rpm -Va file <path> ldd <executable> strace <pid or command> tcpdump -i <if> -w capture.pcap ps -ef netstat -anplt lsof -i iftop, nethogs find / -perm /2000 find / -perm /4000 find / -ctime 1 cat /etc/ld.so.preload echo $LD_PRELOAD cat /etc/ld.so.cache cat /etc/passwd cat /etc/shadow last, lastlog, lastb, who, w strings <binary> | egrep '<pattern>' more /etc/sudoers ls -al /usr/sbin/sshd vim /var/spool/cron/<user>

Cleanup and hardening recommendations

When removing malware, execute all necessary cleanup steps in one operation because many trojans include self-recovery features.
Remove unauthorized SSH keys from all users' .ssh directories except those explicitly required for operations such as bastion access.
Delete suspicious files in /tmp and related temporary directories.
Ensure no accounts have empty passwords and remove unnecessary privileged tools such as socat if not required.
If core system binaries or libraries were replaced, consider rebuilding the system from a trusted image and restoring verified configurations and data.