Linux Commands For Incident Response

There is plenty of volatile data that can be collected from a suspect system. Collecting this data will help you make a preliminary determination as to whether or not there was an incident. Some of the more common pieces of data you should collect are discussed below.

To display the system date and time, including the time zone, enter the following command.

date

To display the name of the system, enter the following command

 hostname

 

System Environment

A concise set of system environment descriptors such as the Operating System, kernel version, and hardware platform can be acquired with the following command. The –a flag stands for all information.

uname -a

A granular snapshot of a subject system environment and status can be obtained by entering the following commands.

printenv
env

Additional version of information with some additional details, such as compiler version is available in the proc/version file.

cat /proc/version

Additional information about the system environment is also available in the /proc directory, including details about the CPU in the /proc/cpuinfo and parameters used to boot the kernel in /proc/cmdline.

cat /proc/cpuinfo
cat /proc/cmdline

Logged In Users 

There are a number of utilities that can be deployed during live response to identify users logged onto a subject system. These commands provide information about accounts that are currently logged into a system by querying the utmp file. The utmp file contains a simple database of active login sessions, with information about the user account, duration, and origin (console or remote host name/IP address) of each session. To determine the current effective users on the system, enter the following commands.

whoami
who
w
logname
id
last
lastb

The id command provide additional details about the current user, including the uid, gid, and which groups the user is in. The last command displays the users who were logged in recently, if the system crashed or was shut down while a user was logged in, and when the system was booted. The lastb command displays a list of failed log in attempts.

The utmp file can become corrupt and report erroneous information. So, when investigating what appears to be suspicious user activity, some effort should be made to confirm that the account of concern is actually logged into the system.

Examination of the /etc/passwd and /etc/shadow files also helps detect suspicious activities in user accounts such as new accounts created by a hacker or unauthorized modifications to existing accounts.

To determine how long the subject system has been running, enter the following command.

uptime

Establishing how long the system has been running gives incident responders a sense of when the system was last rebooted. Knowing that the subject system has not been rebooted since malware was installed can be important, motivating digital investigators to look more closely for deleted processes and other information in memory that otherwise might have been destroyed.

Network Configuration

To obtain information about the network configuration of the subject system, enter the following command.

ifconfig

When documenting the configuration of the subject system, look out for unusual items. Determine whether a network card of the subject system is in promiscuous mode, which generally indicates that a sniffer is running. Look for a VPN adapter configured on a system that does not legitimately use a VPN.

 

Network Connection

Network connections and activity on the subject system can reveal vital information about an attacker’s connection to the system, including the location of an attacker’s remote data collection server and whether the subject system is beaconing to a command and control structure, among other things.

In investigating a potentially infected and compromised system, try to obtain the following information about the network activity on the subject system:

• Active network connections
• Address Resolution Protocol (ARP) cache
• Internet routable table

These details and more can be obtained via the netstat utility. For granularity of results, enter the following command:

netstat -anp
//a -> all sockets
//n -> use numeric IPs (do not resolve host names
//p -> process identifiers and programs

Connections can also be listed using the following command

ss

It is useful to investigate the routing table to detect manipulation by malicious actors. The following command should be entered

netstat -nr

Some malware alter or poison the IP-MAC address relationships in the ARP cache to redirect all network traffic to another computer on the local network that captures the traffic. To investigate the ARP cache, enter the following command

arp -a

Collecting Process Information

During live response, collect certain information pertaining to each process to gain process context, or a full perspective about the process and how it relates to the system state and to other artifacts collected from the system. To gain the broadest perspective, a number of tools gather valuable details relating to processes running on a subject system.

To collect a simple list of running processes and assigned process identifiers (PIDs) from a subject system, including process activity times, enter the following command

ps -ef
ps aux

To examine the amount of system resources that processes are consuming, enter the following command

top

As the top command constantly updates and displays system status in real time (the standard output of which is binary if simply piped to file), capturing the contents to a text file for meaningful analysis can be a challenge. To accomplish this, enter the following command

top -n 1 -b

To obtain detailed system usage, enter the following command

pidstat

The pidstat utility has options to report page faults ( -r), stack utilization ( -s), and I/O statistics ( -d) including the number of bytes written and read per second by a process. This information may be helpful in identifying processes that are logging keystrokes or transferring large amounts of data to/from the compromised system.

To gather resource consumption details for a specific target process, use the –p <target_pid> command option.

pidstat -p 1110

Once a target process has been identified, the location of the associated executable program can be uncovered using the whereis and which commands.

The whereis command locates the source/binary and manual entries for target programs; to query simply for the binary file, use the –b switch. Similarly, the which command shows the full system path of the queried program (or links) in the current environment; no command-line switches are needed. The which –a command displays all matching executable in PATH, not just the first. Suppose, for example, that during initial analysis of running processes on a subject system, a rogue process named logkeys (a GNU/Linux keylogging program) was discovered, the following whereis and which commands will reveal the system path to the associated suspect executable.

whereis -b logkeys
which -a logkeys

Be aware that the which command only searches in locations in the PATH environment variable. So, the PATH environment variable could be modified by an attacker to omit certain directories from a search using the which command.

In identifying the executable program that initiated a process, it is useful to determine the owner of the process to gain user and security context relating to the process. Anomalous system users or escalated user privileges associated with running processes are often indicative of a rogue process. Using the ps aux command can also identify the associated username of a running process. However, to gain a granular context regarding a specific target – real and effective ID – by querying for all processes associated with the username by using the following command:

ps -u <username> //the -u argument is case-insensitive

An alternative command to gather deeper context regarding the owner of a suspect process is

ps -eo pid, user, group, args, etime, lstart | grep '<suspect PID>'

Upon execution, malware spawns additional processes, or child processes. To obtain a structure hierarchical ‘tree’ view of processes, query the suspect system with the ps or pstree utility. The table below provides command options to achieve varying levels of process tree details.

Tool	Command	Description
ps	ps -ejh	Displays the Process ID (PID), Process Group ID (PGID), Session ID (SID), Controlling Terminal (TTY), time the respective processes have been running (TIME), and associated Command-line parameters (COMMAND).
	ps -axjf	Displays the Parent Process ID (PPID), Process ID (PID), PGID, SID, TTY, Process Group ID associated with the controlling TTY Process Group ID (TPGID), Process State (STAT), User ID (UID), TIME, and Command-line parameters (COMMAND).
	ps -aux	Displays the User ID (UID), PID, CPU usage (%CPU), Memory usage (%MEM), Virtual Set Size (VSS), Resident Set Size (RSS), TTY, Process State (STAT), Process start time/date (START), TIME, and COMMAND
pstree	pstree -a	Displays command-line arguments
	pstree -al	Displays command-line arguments using long lines (non-truncated)
	pstree -ah	Displays command-line arguments and highlights each current process and its ancestors.

 

A great utility for viewing the libraries loaded by a running process is pmap. This does not only identify the modules invoked by a process but reveals the memory offset in which the respective libraries have been loaded.

pmap -d <PID>

The command-line parameters associated with target process can be obtained by querying the suspect system with a number of different commands:

ps -eafww
ps -auxww

The ww switch ensures unlimited width in output so that the long command-line arguments are captured.

Correlate Open Ports With Running Processes 

Process-to-Port correlation can be conducted by querying a subject system with a conjunction of the netstat, lsof, and fuser commands.

To determine which process (and associated user) is listening on a specific port using the fuser utility, enter the following command

fuser -u <PID>/<protocol>
//Example
fuser -u 1161/udp

To determine which process is listening on a specific port using the netstat utility, enter the following command

netstat -anp

The executable that is associated with this suspicious process can be found using the lsof command

lsof -p 1161

In addition to providing information about open ports, the fuser command can show which processes are accessing a particular file or directory. To determine which processes (and associated user) are accessing a specific directory (tmp/evil) using the fuser utility, enter the following command.

fuser -u /tmp/evil

Open Files and Dependencies 

Determining the files a particular process has opened can lead a forensic investigator to additional sources of evidence.

The lsof command reveals the files and sockets being accessed by each running program and the username associated with each process.

lsof

The output of lsof also shows which ports and terminals a process has opened. Using the following oprtions provides a list of just the open ports with the associated process and network connections.

lsof -i -n -P

Identify Running Services 

Malware can manifest on a victim system as a service, silently running in the background, unbeknownst to the user. To identify running services, query the subject system with the service command and grep the results for running services (denoted by the “+” symbol)

service --status-all | grep +

Examine Loaded Modules 

Malware may be loaded as a kernel module on the compromised system. Currently loaded modules can be viewed using the lsmod command which displays information that is stored in the /proc/modules file.

lsmod

Collecting The Command History

Commands executed on the compromised computer may be listed in the command history of whatever user account(s) were used. The Bash shell on Linux generally maintains a command history in a file named .bash_history in each user account. Other Linux and UNIX shells store such information in files name .history and .sh_history for each account. If it exists, examine the command history of the account that was used by the intruder. This can be displayed by using the history command.

history

Identifying Mounted and Shared Drives 

To simplify management and backups, rather than storing user files locally, many organizations configure Linux systems to store user home directories, email, and other data remotely on centralized servers.

Information about mounted drives is available in /proc/mounts and /etc/fstab files and the same information is available using the following commands:

df
mount

Conversely, malware can be placed on a system via directories that are shared on the network via Samba, NFS, or other services. Shares exported by the NFS service are configured in the /etc/exports file.

The Samba configuration file, located in the /etc/samba/smb.conf by default, shows any shares that are exported. A review of shares and mounted drives should be reviewed with system administrators to ascertain whether there are any unusual entries.

Determine Scheduled Tasks

Malware may be scheduled to restart periodically in order to persist on a compromised system after reboot. Running the at command will show upcoming scheduled processes, and the associated queue is generally in the /var/spool/cron/at/jobs and /var/spool/at/spool directories.

Examining crontab configuration files on the system will also reveal routine scheduled tasks. In general, Linux systems have a system crontab file (for example, /etc/crontab), and some systems also have daily, hourly, weekly, and monthly configurations (for example, etc/cron.daily, etc/cron.hourly, etc/cron.weekly, and etc/cron.monthly).

In addition, cronjobs can be created with a user account. The queue of jobs that have been scheduled with a specific user account can be found under the var/spool/cron/crontabs in subdirectories for each user account.

Collecting Clipboard Contents

Where the infection vector of a potentially compromised system is unknown, the clipboard contents may provide substantial clues into the nature of an attack, particularly if the attacker is a ‘insider’ and has copied bits of text to paste into tools or attack strings.

Examine the contents of a suspect system’s clipbord using the xclip command as shown below.

xclip -o