Every process is created by forking an existing one, then (usually) replacing its image via exec. PID 1 (systemd or init) is the ancestor of all user processes.
Process states
| State | Code | Meaning |
|---|---|---|
| Running / Runnable | R | On CPU or in the run queue, ready to run. |
| Interruptible sleep | S | Waiting for an event (I/O, signal, timer). Can be woken by a signal. |
| Uninterruptible sleep | D | Waiting for I/O that cannot be interrupted (e.g., disk, NFS). Signals are ignored. High D-state count indicates I/O pressure. |
| Stopped | T | Paused by SIGSTOP or a debugger (ptrace). |
| Zombie | Z | Exited but not yet reaped by parent (wait() not called). Consumes a PID entry only. |
| Dead | X | Being removed; not visible in ps. |
fork / exec / clone
// fork: duplicate the calling process pid_t pid = fork(); if (pid == 0) { // child: execve replaces the process image execve("/bin/ls", args, envp); } else { // parent: wait for child to avoid zombie waitpid(pid, &status, 0); } // clone: low-level syscall behind fork() and pthread_create() // Flags control which resources are shared: // CLONE_VM (share memory), CLONE_FS (share cwd/root), // CLONE_FILES, CLONE_SIGHAND, CLONE_THREAD (same TGID)
Zombie and orphan processes
# Find zombie processes ps aux | awk '$8 == "Z"' ps -eo pid,ppid,stat,comm | grep ' Z' # Find the parent of a zombie and investigate ps -o pid,ppid,stat,comm -p <zombie-pid> # If parent is stuck, killing/restarting it lets init reap zombies # Orphan process — parent died; reparented to PID 1 (init/systemd) cat /proc/<pid>/status | grep PPid
Each process carries several IDs that control its relationship to other processes, its session, and its privileges.
ID types
| ID | Meaning |
|---|---|
| PID | Unique process identifier. |
| PPID | Parent's PID. Used to build the process tree. |
| PGID | Process group ID. Processes in the same group can be signalled together (kill -PGID). |
| SID | Session ID. A session contains one or more process groups. A terminal is associated with one session. |
| TGID | Thread group ID. All threads of a process share the same TGID (= PID of the main thread). |
| UID / EUID | Real and effective user ID. EUID is used for permission checks; can differ via setuid binaries. |
| GID / EGID | Real and effective group ID. |
# Show full process tree with IDs ps -eo pid,ppid,pgid,sid,uid,comm --forest # Show IDs of current shell echo "PID=$$ PPID=$PPID PGID=$(ps -o pgid= $$) SID=$(ps -o sid= $$)" # Show all threads of a process ps -eLf | grep <pid> ls /proc/<pid>/task/ # one dir per thread
/proc is a virtual filesystem exposing kernel data structures as files. Each process has a directory /proc/<PID> with detailed runtime information.
Per-process /proc/PID entries
| Path | Contents |
|---|---|
| /proc/PID/cmdline | Full command line (null-separated args). |
| /proc/PID/environ | Environment variables (null-separated). |
| /proc/PID/status | Human-readable stats: Name, State, Pid, PPid, Threads, VmRSS, … |
| /proc/PID/stat | Machine-readable stats used by ps/top (41+ fields). |
| /proc/PID/maps | Memory mappings — virtual address ranges, permissions, mapped files. |
| /proc/PID/smaps | Extended maps with RSS/PSS/private/shared per region. |
| /proc/PID/fd/ | Symlinks to every open file descriptor. |
| /proc/PID/fdinfo/ | Flags and position for each fd. |
| /proc/PID/net/ | Network state from the process's network namespace. |
| /proc/PID/cgroup | cgroup membership for each hierarchy. |
| /proc/PID/oom_score | Current OOM killer score (higher = more likely to be killed). |
| /proc/PID/oom_score_adj | Writable bias (-1000 to 1000) added to oom_score. |
# Command line of a process tr '\0' ' ' < /proc/<pid>/cmdline; echo # Memory usage breakdown grep -E 'VmRSS|VmSize|VmSwap|Threads' /proc/<pid>/status # Open files ls -la /proc/<pid>/fd # Find deleted files held open (common cause of disk-full) ls -la /proc/*/fd 2>/dev/null | grep deleted # Memory maps cat /proc/<pid>/maps # Stack trace (if kernel supports it) cat /proc/<pid>/stack
Signals are asynchronous notifications sent to a process. The kernel delivers them by interrupting the process between instructions. Each signal has a default action; processes can override it with a handler (except SIGKILL and SIGSTOP).
Common signals
| Signal | No. | Default action | Common use |
|---|---|---|---|
| SIGHUP | 1 | Terminate | Reload config (daemons often catch this) |
| SIGINT | 2 | Terminate | Ctrl+C — interrupt from terminal |
| SIGQUIT | 3 | Core dump | Ctrl+\ — quit with core dump |
| SIGILL | 4 | Core dump | Illegal instruction |
| SIGTRAP | 5 | Core dump | Debugger breakpoint (ptrace) |
| SIGABRT | 6 | Core dump | abort() — assertion failure |
| SIGBUS | 7 | Core dump | Misaligned memory access |
| SIGFPE | 8 | Core dump | Floating-point / division by zero |
| SIGKILL | 9 | Terminate | Cannot be caught or ignored |
| SIGSEGV | 11 | Core dump | Invalid memory access |
| SIGPIPE | 13 | Terminate | Write to closed pipe/socket |
| SIGALRM | 14 | Terminate | alarm() timer expiry |
| SIGTERM | 15 | Terminate | Graceful shutdown request (default of kill) |
| SIGCHLD | 17 | Ignore | Child state change (exit/stop) |
| SIGCONT | 18 | Continue | Resume a stopped process |
| SIGSTOP | 19 | Stop | Cannot be caught or ignored |
| SIGTSTP | 20 | Stop | Ctrl+Z — terminal stop (catchable) |
| SIGUSR1/2 | 10/12 | Terminate | User-defined; application-specific semantics |
# Send SIGTERM (graceful shutdown) — default kill <pid> kill -15 <pid> kill -TERM <pid> # Force kill — use only if SIGTERM fails after a timeout kill -9 <pid> kill -KILL <pid> # Reload config kill -HUP <pid> # Send to whole process group (negative PID) kill -TERM -<pgid> # By name pkill nginx # SIGTERM to all matching pkill -9 -x nginx # exact name match, SIGKILL pkill -HUP -u www-data nginx # match user + name # killall — by exact name killall -TERM gunicorn # Show all signal names kill -l
trap in shell scripts
cleanup() { echo "cleaning up..."; rm -f /tmp/my-lockfile; }
trap cleanup EXIT # always run on exit
trap cleanup INT TERM # also on Ctrl+C and kill
trap '' HUP # ignore SIGHUP
# Run in background my-command & jobs # list background jobs fg %1 # bring job 1 to foreground bg %1 # resume stopped job in background # Ctrl+Z suspends foreground process (sends SIGTSTP) # disown — detach job from shell (won't get SIGHUP when shell exits) my-command & disown %1 # nohup — immune to SIGHUP; redirects stdout/stderr nohup my-command > /var/log/my.log 2>&1 & # setsid — create new session (fully detach from terminal) setsid my-command & # Run in new process group set -m # enable monitor mode (job control) in scripts
Resource limits (rlimits) cap per-process resource consumption. Each limit has a soft value (current enforcement) and a hard value (ceiling a process can raise its soft limit to).
Common limits
| Limit | ulimit flag | Default | Common fix |
|---|---|---|---|
| Open files | -n | 1024 | Raise to 65536+ for servers/databases |
| Max processes | -u | varies | Raise for apps that fork heavily |
| Stack size | -s | 8MB | Raise for deep recursion; lower for thread safety |
| Core file size | -c | 0 (disabled) | Set to unlimited for debugging crashes |
| Virtual memory | -v | unlimited | Cap for memory-leaking processes |
# Show current limits ulimit -a # Raise open file limit for current shell session ulimit -n 65536 # Check limits of a running process cat /proc/<pid>/limits # Persistent limits in /etc/security/limits.conf * soft nofile 65536 * hard nofile 131072 www-data soft nproc 2048 # Set limits when launching a process (prlimit) prlimit --nofile=65536:131072 -- my-server # Set limits in a systemd service # [Service] # LimitNOFILE=65536 # LimitNPROC=4096
Control groups (cgroups) limit, account for, and isolate resource usage of process groups. cgroups v2 uses a unified hierarchy (one tree, multiple controllers). systemd uses cgroups v2 slices to enforce per-service limits.
Key controllers
| Controller | Limits |
|---|---|
| cpu | cpu.weight (relative shares), cpu.max (quota/period for hard cap) |
| memory | memory.max (hard limit), memory.high (soft throttle), memory.swap.max |
| io | io.max (IOPS/bandwidth per device), io.weight |
| pids | pids.max (prevent fork bombs) |
# Inspect cgroup hierarchy systemd-cgls cat /proc/<pid>/cgroup # Show resource usage for a service systemctl status my-service # shows CPU/memory at bottom systemd-cgtop # live per-cgroup resource usage # Set memory limit on a service (systemd) systemctl set-property my-service MemoryMax=512M # Or in the unit file [Service] section: # MemoryMax=512M # CPUQuota=50% # TasksMax=100 # Create a cgroup manually (v2) mkdir /sys/fs/cgroup/my-group echo "512M" > /sys/fs/cgroup/my-group/memory.max echo <pid> > /sys/fs/cgroup/my-group/cgroup.procs
Process inspection
ps aux --forestpstree -ptop / htopcat /proc/<pid>/status
Signals
kill -TERM <pid> — gracefulkill -9 <pid> — forcekill -HUP <pid> — reloadpkill -f pattern
/proc quick hits
tr '\0' ' ' < /proc/PID/cmdlinels /proc/PID/fd | wc -l — fd countcat /proc/PID/limitscat /proc/PID/oom_score
Resource limits
ulimit -n 65536 — open filescat /proc/PID/limitsprlimit --nofile=65536 -- cmd
Persistent: /etc/security/limits.conf
Job control
Ctrl+Z suspend, bg resumenohup cmd & — HUP immunedisown %1 — detach from shellsetsid cmd — new session
cgroups v2
systemd-cglssystemd-cgtopsystemctl set-property svc MemoryMax=1Gcat /proc/PID/cgroup