Linux Processes & Job Management Tutorial: Master ps, top, kill, jobs, and Server Troubleshooting

This Chapter introduces the essential Linux commands used to view, monitor, find, and manage running processes. In Linux, every running program is a process, and each process has a unique Process ID (PID). Before you can stop, prioritize, or troubleshoot a process, you must first know how to identify it correctly.

ps - Displays a snapshot of running processes

Syntax

ps [options]

Important options

Beginner example

ps
ps aux | head -n 10
ps -ef --forest | head -n 20

Practical server-administration examples

Highest CPU consumers:

ps aux --sort=-%cpu | head -n 15

Highest memory consumers:

ps aux --sort=-%mem | head -n 15

Inspect one process:

ps -p 1234 -o pid,ppid,user,stat,%cpu,%mem,etime,cmd

Inspect a user’s processes:

ps -u alice -o pid,ppid,stat,%cpu,%mem,cmd

Inspect a service:

ps -C nginx -o pid,ppid,user,stat,cmd

Reading output

USER  PID  %CPU %MEM STAT COMMAND
alice 4821 92.3 1.2 R    python3 cpu_load.py

Verification

ps -p "$pid" -o pid,ppid,user,stat,%cpu,%mem,etime,cmd

top - Monitors processes and system resources in real time

Syntax

top [options]

Important options

Interactive keys

Beginner example

top
# Press q to exit.

Practical server-administration examples

Monitor one process:

top -p 1234

Take one noninteractive snapshot:

top -b -n 1 | head -n 25

Monitor a user:

top -u alice

Reading top output

Common Mistakes

Verification

top -b -n 1 -p "$pid"

htop - Provides an enhanced interactive process monitor with easier navigation, filtering, sorting, and tree views.

Install

Ubuntu/Debian:

sudo apt update
sudo apt install htop

RHEL/Rocky/Alma/Fedora:

sudo dnf install htop

Important keys

Beginner example

htop

Practical server-administration examples

htop -u alice
htop -p 1234

When htop is better

htop is a monitoring aid, not a substitute for ps, logs, metrics, and incident evidence.

Common mistakes

Verification

ps -p "$pid" -o pid,user,stat,%cpu,%mem,cmd

pstree - Displays the process hierarchy

Syntax

pstree [options] [PID or user]

Important options

Beginner example

pstree -p "$$"

Practical server-administration example

Inspect the ancestry of a workload:

pid=$(pgrep -f 'cpu_load.py' | head -n 1)
pstree -ap "$pid"

Common mistakes

Verification

ps -ef --forest

pgrep - Finds process IDs by name, user, parent, or full command line

Syntax

pgrep [options] pattern

Important options

Beginner example

pgrep -a sleep

Practical server-administration example

Find a specific workload before acting:

pgrep -af "python3 cpu_load.py"

Why pgrep is safer

pgrep matches process attributes directly and avoids accidental matching of the grep command itself.

Compare:

ps aux | grep python

with:

pgrep -af python3

Common mistakes

Verification

pgrep -af "cpu_load.py"
ps -fp "$(pgrep -f 'cpu_load.py' | tr '\n' ',' | sed 's/,$//')"

kill - Sends a signal to a process. It does not always terminate it

Syntax

kill [options] PID

Important signals

Examples

kill 1234
kill -15 1234
kill -TERM 1234
kill -9 1234
kill -KILL 1234
kill -STOP 1234
kill -CONT 1234
kill -l

Safety rules

Common mistakes

Verification

kill -0 1234 2>/dev/null && echo "Process exists" || echo "Process absent"
pgrep -af "cpu_load.py"

pkill - Sends signals to processes selected by name, user, parent, or command-line pattern

Syntax

pkill [options] pattern

Important options

Beginner example

pgrep -l nginx
pkill -TERM nginx

Practical server-administration example

Terminate a specific controlled workload:

pgrep -af "python3 cpu_load.py"
pkill -TERM -f "python3 cpu_load.py"

Safety warning

pkill can match multiple processes.

Avoid:

pkill python
pkill java
pkill bash

Always inspect first:

pgrep -af pattern

Common mistakes

Verification

pgrep -af "cpu_load.py"

killall - Sends signals to processes with an exact command name

Syntax

killall [options] NAME

Important options

Beginner example

killall -i firefox

Practical server-administration example

killall -TERM nginx

Prefer systemctl restart nginx when the service is managed by systemd.

Difference

Common mistakes

Verification

pgrep -x nginx

jobs - Displays jobs managed by the current shell

Syntax

jobs [options]

Important options

Beginner example

sleep 300 &
jobs
jobs -l
jobs -r
jobs -s

Practical server-administration example

Check whether a backup job is still running before closing a maintenance terminal:

jobs -l

Common mistakes

Verification

jobs -l

fg and bg - Move shell jobs between foreground and background

Syntax

fg [%JOB]
bg [%JOB]

Beginner example

sleep 300

Press:

Ctrl+Z

Then:

jobs
bg %1
fg %1

Job ID versus PID

jobs -l

Terminal closure

A normal background job may receive SIGHUP when its controlling terminal closes. To keep it running:

nohup ./long_task.sh > long-task.log 2>&1 &
disown -h %1

nohup ignores hangup signals. disown -h prevents the shell from sending SIGHUP to the job when the shell exits.

Common mistakes

Verification

jobs -l
ps -fp "$(jobs -p | tr '\n' ',' | sed 's/,$//')"

nice - Starts a process with a scheduling priority

Syntax

nice [options] [command]

Nice values

Higher nice value means lower scheduling priority. Lower nice value means higher scheduling priority.

Only root can normally set a negative nice value.

Important options

Beginner example

nice -n 10 sleep 300 &
ps -o pid,ni,cmd -p "$!"

Practical server-administration examples

Run a backup at low priority:

nice -n 19 ./backup.sh

Run a batch report without competing aggressively for CPU:

nice -n 10 python3 report.py

Start an important service with higher priority:

sudo nice -n -5 /usr/local/bin/important_service

Common mistakes

Verification

ps -o pid,ni,cmd -p "$!"

renice - Changes the nice value of an already-running process

Syntax

renice [options] priority [-p PID] [-u USER]

Important options

Beginner example

sleep 300 &
pid=$!
renice -n 10 -p "$pid"
ps -o pid,ni,cmd -p "$pid"

Practical server-administration example

Reduce the scheduling priority of a runaway report process:

pid=$(pgrep -f 'python3 report.py' | head -n 1)
renice -n 19 -p "$pid"

Increase priority only when authorized:

sudo renice -n -5 -p "$pid"

Common mistakes

Verification

ps -o pid,ni,cmd -p "$pid"

This Chapter 3 explains how to identify and understand the CPU, memory, disk I/O, and system-load usage of Linux processes. It helps you find which process is consuming excessive resources, decide whether that usage is normal, and investigate issues such as slow servers, high load, swap pressure, and unexpected service restarts.

CPU usage - Measuring processor consumption

CPU usage measures how much processor time a process consumes. High CPU is not automatically bad; a backup, compiler, or report may legitimately use CPU.

Find high-CPU processes

ps aux --sort=-%cpu | head -n 15
top -b -n 1 | head -n 20

Memory usage - Physical, virtual, and swap metrics

Memory concepts

Check system memory

free -h
cat /proc/meminfo | head -n 20

Find high-memory processes

ps aux --sort=-%mem | head -n 15

Inspect one process

pid=$(pgrep -f 'cpu_load.py' | head -n 1)
cat "/proc/$pid/status" | grep -E '^(Name|State|Pid|PPid|VmRSS|VmSize|Threads)'

Disk I/O and load - Uncovering non-CPU bottlenecks

Disk I/O can cause high load without high CPU.

vmstat 1 5
uptime
du -sh /var/log

Example vmstat output

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 8123456 123456 2345678    0    0    12    45  120  240  8  2 88  2  0

High wa means significant I/O wait. High b can indicate blocked processes.

OOM killer and limits - Kernel terminations and resource boundaries

When memory is exhausted, the kernel may invoke the OOM killer and terminate a process. This can appear as an unexpected service restart.

Check logs (Ubuntu/Debian)

sudo journalctl -k | grep -i -E 'oom|out of memory' | tail -n 50
sudo grep -i -E 'oom|out of memory' /var/log/kern.log 2>/dev/null | tail -n 50

Check logs (RHEL-family)

sudo journalctl -k | grep -i -E 'oom|out of memory' | tail -n 50
sudo grep -i -E 'oom|out of memory' /var/log/messages 2>/dev/null | tail -n 50

Inspect process limits

pid=$(pgrep -f 'cpu_load.py' | head -n 1)
cat "/proc/$pid/limits"
ulimit -a

Identify problem processes - Rapid diagnostic commands

Signals in Depth explains how Linux communicates with running processes using signals. A signal is a notification sent to a process to interrupt it, pause it, resume it, reload its configuration, or request it to terminate.

Linux Signals - Inter-Process Notifications and Control

Signals are notifications sent to processes.

SIGINT versus SIGTERM versus SIGKILL - Termination Comparison

SIGSTOP and SIGCONT - Freezing and Resuming Workloads

Pause a process for investigation:

kill -STOP 1234
ps -p 1234 -o pid,stat,cmd

Resume it:

kill -CONT 1234

SIGSTOP is useful for temporarily freezing a noncritical process, but do not leave a production service stopped.

Sending signals - Common Command Invocations

kill -TERM PID
pkill -TERM -f "python3 cpu_load.py"
killall -TERM process-name

In top, press k, enter the PID, then choose a signal.

This Chapter explains three important types of Linux processes: daemons, zombie processes, and orphan processes. Understanding them helps you identify normal background services, detect failed child-process handling, and investigate unexpected or long-running processes on a Linux server.

Daemons - Background Service Architecture

Daemons are background service processes.

Inspect common services

systemctl status ssh 2>/dev/null || systemctl status sshd 2>/dev/null || true
systemctl status nginx 2>/dev/null || true
ps -C sshd -o pid,ppid,user,stat,cmd
pgrep -a sshd
pgrep -a nginx

Ubuntu commonly uses ssh.service; RHEL-family systems commonly use sshd.service.

Zombie processes - Dead Processes Awaiting Parent Reaping

Find zombies

ps aux | awk '$8 ~ /^Z/ {print}'
ps -eo pid,ppid,stat,cmd | awk '$3 ~ /^Z/ {print}'

Inspect the parent

zombie_pid=$(ps -eo pid,stat --no-headers | awk '$2 ~ /^Z/ {print $1; exit}')
if [[ -n "$zombie_pid" ]]; then
  ppid=$(ps -o ppid= -p "$zombie_pid" | tr -d ' ')
  ps -fp "$ppid"
fi

Fix the parent or restart the affected application. Killing the zombie itself does not solve the underlying reaping problem.

Orphan processes - Reparented Processes

Find long-running processes

ps -eo pid,ppid,user,stat,etime,cmd --sort=-etime | head -n 20

An orphan may be normal if it is a daemon or detached job. Investigate if it is unexpected, privileged, or consuming resources.

Long-running processes - Monitoring Execution Durations

Check elapsed time

ps -eo pid,user,etime,cmd --sort=-etime | head -n 20

Example output

  PID USER     ELAPSED COMMAND
  987 root     12-03:22:11 /usr/sbin/sshd -D
 1234 student 01:22:10 python3 process-lab/signal_demo.py

Real-World Server Use Cases applies Linux process and job management skills to practical server administration situations. Instead of only learning commands, this section shows how to investigate and respond safely to common issues such as high CPU usage, high memory consumption, unresponsive applications, suspicious processes, and long-running background jobs.

High CPU usage - Step-by-Step Triage

1. Identify the process:

ps aux --sort=-%cpu | head -n 15

2. Confirm ownership and command:

pid=$(ps aux --sort=-%cpu | awk 'NR==2 {print $2}')
ps -fp "$pid"
pstree -ap "$pid"

3. Determine whether it is expected.

4. Reduce priority if safe:

renice -n 19 -p "$pid"

5. Stop or terminate only after approval:

kill -TERM "$pid"

6. Restart the affected service if needed:>

sudo systemctl restart service-name

High memory usage - Identification and Mitigation

1. Identify the consumer:>

ps aux --sort=-%mem | head -n 15
free -h

2. Inspect memory details:

pid=$(ps aux --sort=-%mem | awk 'NR==2 {print $2}')
grep -E '^(Name|State|VmRSS|VmSize|Threads)' "/proc/$pid/status"

3. Check swap and OOM logs:

free -h
sudo journalctl -k | grep -i -E 'oom|out of memory' | tail -n 50

4. Review application logs and metrics.>

5. Restart only when approved and safe:

sudo systemctl restart service-name

Unresponsive application - Diagnostic and Termination Escalation

1. Check state:

pid=$(pgrep -f 'application-name' | head -n 1)
ps -p "$pid" -o pid,ppid,user,stat,%cpu,%mem,etime,cmd

2. Send SIGTERM:

kill -TERM "$pid"

3. Wait and verify:

sleep 10
pgrep -af 'application-name'

4. Escalate to SIGKILL only if required:

kill -KILL "$pid"

5. Restart through systemd:

sudo systemctl restart service-name

Suspicious process - Evidence Collection and Forensics

Collect evidence before killing:

pid=$(pgrep -f 'suspicious-pattern' | head -n 1)
ps -fp "$pid"
cat "/proc/$pid/cmdline"; printf '\n'
readlink -f "/proc/$pid/exe"
ls -l "/proc/$pid/exe"
pstree -ap "$pid"

Check network activity if authorized tools are installed:

sudo ss -tunap | grep "pid=$pid"

Background job management - Execution, Monitoring, and Detachment

Start a long backup:

nohup ./backup.sh > backup.log 2>&1 &
disown -h %1

Check it:

jobs -l
pgrep -af backup.sh
tail -f backup.log

Monitor CPU:

pid=$(pgrep -f 'backup.sh' | head -n 1)
top -p "$pid"

The Hands-On Labs section gives you practical experience with Linux process and job management in a safe, controlled environment. Instead of only reading commands, you will create test processes, inspect their PID, PPID, state, CPU, and memory usage, and practice monitoring them with tools such as ps, top, htop, pgrep, and pstree.

The labs include Process Investigation, Signal Management, and High CPU Simulation. Each lab has a clear objective, step-by-step tasks, commands, expected results, and a verification checklist so you can confirm that you understand both the command and the reason behind it.

Lab 7.1 - Process Investigation

Objective

Identify PID, PPID, CPU usage, memory usage, command, and process state.

Task

Solution

In terminal 1:

sleep 600

In terminal 2:

pgrep -a sleep

Example output:

4821 sleep 600

Inspect it:

pid=$(pgrep -f '^sleep 600$' | head -n 1)
ps -p "$pid" -o pid,ppid,user,stat,%cpu,%mem,etime,cmd

Example output; your values will differ:

    PID    PPID USER     STAT %CPU %MEM      ELAPSED COMMAND
   4821    1234 student  S     0.0  0.0       00:01:12 sleep 600

View the process tree:

pstree -p "$pid"
pstree -p

Monitor it:

top -p "$pid"

Inspect /proc:

cat "/proc/$pid/status"
cat "/proc/$pid/cmdline"; printf '\n'
ls "/proc/$pid"

Record:

PID: 4821
PPID: 1234
CPU: near 0%
Memory: near 0%
Command: sleep 600
State: S
Parent: interactive Bash shell

Expected-results checklist

Lab 7.2 - Signal Management

This Lab helps you understand how Linux signals control a process. Using a small Python script, you practice interrupting a foreground process with Ctrl+C (SIGINT), pausing it with SIGSTOP, resuming it with SIGCONT, and terminating it gracefully with SIGTERM.

Objective

Practice SIGINT, SIGTERM, SIGKILL, SIGSTOP, and SIGCONT safely.

Task

Create a Python test process, then pause, resume, terminate, and verify it.

Solution

Create the script:

cd ~/process-lab
cat > signal_demo.py <<'EOF'
import time
while True:
    print("Process running...", flush=True)
    time.sleep(2)
EOF

Run it in the foreground:

python3 signal_demo.py

Press:

Ctrl+C

Ctrl+C sends SIGINT.

Start it in the background:

python3 signal_demo.py > signal-demo.log 2>&1 &

Find its PID:

pid=$(pgrep -f 'python3 signal_demo.py' | head -n 1)
echo "$pid"

Pause it:

kill -STOP "$pid"

Verify:

ps -p "$pid" -o pid,stat,cmd

Expected state:

T

Resume it:

kill -CONT "$pid"
ps -p "$pid" -o pid,stat,cmd

Expected state:

S

Send graceful termination:

kill -TERM "$pid"
sleep 2
pgrep -af 'signal_demo.py' || echo "Process terminated"

If it remains, force kill:

kill -KILL "$pid"
pgrep -af 'signal_demo.py' || echo "Process removed"

Expected-results checklist

Lab 7.3 - High CPU Simulation

This Lab simulates a controlled high-CPU workload so you can practice real server troubleshooting. You run a Python script that creates two CPU-intensive child processes, then identify the parent and child processes using top, htop, ps, pgrep, and pstree.

Objective

Create a controlled CPU workload, investigate it, reduce its priority, and terminate it safely.

Task

Run a two-process CPU workload, identify it, renice it, terminate it, and verify CPU recovery.

Solution

Create the workload:

cd ~/process-lab
cat > cpu_load.py <<'EOF'
import multiprocessing
import time

def burn_cpu():
    while True:
        pass

if __name__ == "__main__":
    processes = []
    for _ in range(2):
        p = multiprocessing.Process(target=burn_cpu)
        p.start()
        processes.append(p)

    time.sleep(120)

    for p in processes:
        p.terminate()
EOF

Run it:

python3 cpu_load.py > cpu-load.log 2>&1 &

Investigate:

top
htop
ps aux --sort=-%cpu | head -n 15
pgrep -af cpu_load.py
pstree -p | grep -A2 -B2 cpu_load

Identify the parent:

parent_pid=$(pgrep -f '^python3 cpu_load.py$' | head -n 1)
echo "Parent PID: $parent_pid"
pgrep -P "$parent_pid" -a

Reduce priority:

renice -n 19 -p "$parent_pid"
ps -o pid,ni,stat,%cpu,cmd -p "$parent_pid"

Terminate gracefully:

kill -TERM "$parent_pid"
sleep 3
pgrep -af cpu_load.py || echo "Workload stopped"

If needed, force kill remaining child processes:

for pid in $(pgrep -f 'cpu_load.py'); do
  kill -KILL "$pid"
done

Verify:

pgrep -af cpu_load.py || echo "No cpu_load processes remain"
top -b -n 1 | head -n 15

Expected-results checklist

Process-management command cheat sheet

ps
ps aux
ps -ef
ps -ef --forest
ps aux --sort=-%cpu | head -n 15
ps aux --sort=-%mem | head -n 15
ps -p PID -o pid,ppid,user,stat,%cpu,%mem,etime,cmd
ps -u USER -o pid,ppid,stat,%cpu,%mem,cmd
ps -C NAME -o pid,ppid,user,stat,cmd

top
top -d 2
top -p PID
top -u USER
top -b -n 1

htop
htop -u USER
htop -p PID

pstree
pstree -p
pstree -ap PID

pgrep NAME
pgrep -l NAME
pgrep -af PATTERN
pgrep -u USER NAME
pgrep -P PPID
pgrep -x NAME

kill PID
kill -TERM PID
kill -KILL PID
kill -STOP PID
kill -CONT PID
pkill -TERM -f PATTERN
killall -TERM NAME

jobs
jobs -l
fg %JOB
bg %JOB
nohup COMMAND > output.log 2>&1 &
disown -h %JOB

nice -n 10 COMMAND
renice -n 19 -p PID
sudo renice -n -5 -p PID

Signal cheat sheet

Process state cheat sheet

Process priority cheat sheet

Troubleshooting - Diagnostic Reference Matrix

Practice Exercises

Interview Questions

1. What is the difference between a program and a process?

A program is a stored executable file. A process is a running instance of that program with a PID, memory, state, and execution context.

2. Why is SIGTERM preferred over SIGKILL?

SIGTERM allows the application to shut down cleanly, flush data, close connections, and release resources. SIGKILL cannot be caught and may leave incomplete work or stale state.

3. What is a zombie process?

A zombie is a terminated process whose parent has not collected its exit status. It consumes a PID entry but not CPU or normal memory. Investigate the parent.

4. What does a high load average mean?

It indicates work waiting for CPU or potentially blocked on uninterruptible I/O. Interpret it relative to CPU count and current process states.

5. Why should pkill be used carefully?

It can match multiple processes. Always inspect matches with pgrep -af before sending a signal, and prefer an exact PID for critical services.