Processes & jobs

What is a process?

Every program that runs on your computer is a process — a running instance of an executable, with its own memory, file handles, and state. The operating system tracks each one with a unique integer called a PID (process identifier). PIDs are assigned at birth and recycled after death; the same number may belong to different programs at different times.

You can have many processes from the same program simultaneously. Running python train.py twice creates two separate processes, each with its own PID.

Seeing what is running

ps aux — a snapshot

ps prints a snapshot of the current process table. The flags aux are nearly universal:

ps aux

USER       PID  %CPU %MEM      VSZ    RSS   STAT STARTED      TIME COMMAND
alice      412   0.0  0.1    33456   4096   Ss   09:01     0:00.03 /bin/zsh
alice     8821  98.2  4.7  3124000 386000   R    10:45     6:14.22 python train.py
alice     8900   0.0  0.0    33000   2048   S+   10:46     0:00.01 ps aux

Key columns:

Filter with grep when you know what you are looking for:

ps aux | grep python

alice     8821  98.2  4.7  3124000 386000   R    10:45     6:14.22 python train.py
alice     8923   0.0  0.0    33000   1024   S+   10:46     0:00.00 grep python

The second line is the grep command itself — a harmless artifact you can ignore.

top and htop — live views

top gives you a live, auto-refreshing view:

top

Press q to quit. The process consuming the most CPU floats to the top automatically.

If htop is installed (it usually is on Linux; brew install htop on macOS), prefer it — it adds color, mouse support, and easier sorting. Press F10 or q to exit.

Process states

A process is not just “running” or “not running.” The table below shows the states you will encounter:

The diagram below shows how a process moves between the states you control from the shell:

Foreground vs background

When you run a command normally, it runs in the foreground: it owns your terminal, you see its output, and you cannot type another command until it finishes.

To run a command in the background from the start, append &:

python train.py &

[1] 8821

The shell prints the job number (in brackets) and the PID. The job number is a shell-local shorthand; the PID is the system-wide identifier.

Moving a running process to the background

If you forgot & and the program is already running in the foreground:

1. Press Ctrl+Z — this sends SIGTSTP, which suspends (freezes) the process and gives you your prompt back.
2. Type bg to resume it in the background.

bg

[1]+ python train.py &

Listing your jobs

jobs

[1]-  Running    python train.py &
[2]+  Stopped    vim notes.txt

The + marks the most recent job; - marks the one before it.

Bringing a job back to the foreground

fg %1

%1 means job number 1. Just fg with no argument brings back the most recent job.

Stopping a process with Ctrl+C

While a process is in the foreground, Ctrl+C sends it SIGINT (signal number 2). Most programs treat this as a polite request to stop immediately. A well-written script catches SIGINT and exits cleanly — flushing buffers, closing files, removing lock files.

Sending signals with kill

Signals are messages the operating system delivers to processes. kill is the command that sends them — despite its name, it can send any signal, not just termination ones.

SIGTERM — the polite request

kill 8821

With no flag, kill sends SIGTERM (signal 15). The process receives the signal and is expected to clean up and exit gracefully. A training script that catches SIGTERM can save a checkpoint before shutting down.

SIGKILL — the forceful hammer

kill -9 8821

SIGKILL (signal 9) cannot be caught, blocked, or ignored. The kernel terminates the process immediately, with zero opportunity for cleanup.

Finding the right PID

If you do not know the PID, use pgrep to look it up by name:

pgrep -l python

8821 python

-l prints the name alongside the PID. To kill by name in one step, use pkill:

pkill python

This sends SIGTERM to every process whose name matches. Add -9 for SIGKILL, and -x to match the name exactly (avoiding collateral kills of unrelated processes).

Surviving logout with nohup

Background jobs (&) are still children of your shell. When you log out or close the terminal, the shell sends SIGHUP to all its children, which usually kills them.

nohup (“no hang-up”) makes a process immune to SIGHUP:

nohup python train.py > train.log 2>&1 &

[1] 9100

Output that would have gone to the terminal is redirected to train.log (or nohup.out if you omit the redirection). The process keeps running after you log out.

For long-running work, consider tmux or screen instead. They create a persistent terminal session you can detach from and reattach to later — even from a different machine:

tmux new -s training
python train.py

Press Ctrl+B then D to detach. The session (and your script) keeps running. Reattach later with tmux attach -t training.

Putting it all together

Here is a realistic workflow for a runaway training script:

ps aux | grep train.py

alice     8821  99.1  6.2  4200000 510000   R    10:45    42:11.00 python train.py

kill 8821

Wait a few seconds. If it is still there:

kill -9 8821

To run a new training job that survives logout and logs output:

nohup python train.py --epochs 100 > run.log 2>&1 &
echo "PID is $!"

$! is a special shell variable that expands to the PID of the last background command — handy to save so you can kill it later.