Pipes & Redirection

You already know commands like grep, ls, and ps. The real power of the command line is not any individual tool — it is connecting them. A single pipeline can replace a custom script, and once it clicks, you will wonder how you ever worked without it.

The three I/O streams

Every process that runs on a Unix-like system has three standard streams attached to it automatically:

The critical insight: stdout and stderr both appear on your screen by default, but they are different channels. That distinction matters as soon as you start redirecting.

The pipe operator

The pipe | connects the stdout of one command to the stdin of the next. The second command never knows or cares where its input came from — it just reads from stdin as usual.

Here is the motivating example from the question above:

ps aux | grep python | wc -l

7

What just happened, step by step:

1. ps aux — list every running process with details, write to stdout
2. grep python — read that list from stdin, pass through only lines containing “python”, write matches to stdout
3. wc -l — count the lines it receives on stdin, print the count

No temporary files. No custom script. Three independent tools, composed.

Redirection: sending streams to files

Instead of displaying stdout on your terminal, you can send it to a file.

Overwrite with >

ls -lh ~/Documents > file-list.txt

The file file-list.txt is created (or erased and replaced if it already exists) with the output.

Append with >>

echo "Run at $(date)" >> run-log.txt

Each execution adds a new line. Nothing is destroyed.

Feed a file into stdin with <

sort < unsorted-names.txt

sort reads the file as if you had typed its contents. This form is less common — most commands accept a filename as an argument — but it matters when a command only reads from stdin.

Redirecting stderr

Remember: errors go to stderr (stream 2), not stdout (stream 1). They will not travel down a pipe unless you redirect them.

Redirect stderr to a file with 2>

find / -name "secrets.txt" 2> errors.txt

Permission-denied messages go to errors.txt; real results still appear on your terminal.

Merge stderr into stdout with 2>&1

./build.sh > build.log 2>&1

Read this right-to-left: “redirect stderr (2) to wherever stdout (1) is currently going” — which is build.log. Now both streams land in one file.

A shorthand that means the same thing:

./build.sh &> build.log

Throw stderr away entirely

find / -name "secrets.txt" 2> /dev/null

/dev/null is a special file that discards everything written to it. Useful when you care about results but not error noise.

Stderr does not travel down a pipe by default

find / -name "*.py" | grep src

Only the found filenames (stdout) flow into grep. The flood of “Permission denied” errors go straight to your terminal on stderr. To pipe everything — results and errors alike:

find / -name "*.py" 2>&1 | grep src

A realistic multi-step example

You want to audit which Python processes are consuming the most memory, save the top 5 to a file, and discard any errors:

ps aux 2>/dev/null | grep '[p]ython' | sort -k4 -rn | head -5 > python-top.txt

shreyash  1234  2.1 12.3 python train.py
shreyash  5678  1.8  9.1 python serve.py
...

Breaking it down:

• ps aux 2>/dev/null — list processes, discard any errors
• grep '[p]ython' — keep Python lines (the bracket trick prevents grep itself from matching)
• sort -k4 -rn — sort by column 4 (memory %) in reverse numeric order
• head -5 — keep the top 5 lines
• > python-top.txt — save the result, overwriting any previous file

Quick reference