Pipes & Redirection

The last lesson promised this exact idea: every command does one small thing, and the power comes from chaining them. Here is the connector that does the chaining.

You already know commands like grep, ls, and ps. The real power of the command line is not any individual tool — it is connecting them. A single pipeline can replace a custom script, and once it clicks, you will wonder how you ever worked without it.

The three I/O streams

Every process that runs on a Unix-like system has three standard streams attached to it automatically:

The critical insight: stdout and stderr both appear on your screen by default, but they are different channels. That distinction matters as soon as you start redirecting.

The pipe operator

The pipe | connects the stdout of one command to the stdin of the next. The second command never knows or cares where its input came from — it just reads from stdin as usual.

Here is the motivating example from the question above:

ps aux | grep python | wc -l

7

What just happened, step by step:

1. ps aux — list every running process with details, write to stdout
2. grep python — read that list from stdin, pass through only lines containing “python”, write matches to stdout
3. wc -l — count the lines it receives on stdin, print the count

No temporary files. No custom script. Three independent tools, composed.

Redirection: sending streams to files

Instead of displaying stdout on your terminal, you can send it to a file.

Overwrite with >

ls -lh ~/Documents > file-list.txt

The file file-list.txt is created (or erased and replaced if it already exists) with the output.

Append with >>

echo "Run at $(date)" >> run-log.txt

Each execution adds a new line. Nothing is destroyed.

Feed a file into stdin with <

sort < unsorted-names.txt

sort reads the file as if you had typed its contents. This form is less common — most commands accept a filename as an argument — but it matters when a command only reads from stdin.

Redirecting stderr

Remember: errors go to stderr (stream 2), not stdout (stream 1). They will not travel down a pipe unless you redirect them.

Redirect stderr to a file with 2>

find / -name "secrets.txt" 2> errors.txt

Permission-denied messages go to errors.txt; real results still appear on your terminal.

Merge stderr into stdout with 2>&1

./build.sh > build.log 2>&1

Read this right-to-left: “redirect stderr (2) to wherever stdout (1) is currently going” — which is build.log. Now both streams land in one file.

A shorthand that means the same thing:

./build.sh &> build.log

Throw stderr away entirely

find / -name "secrets.txt" 2> /dev/null

/dev/null is a special file that discards everything written to it. Useful when you care about results but not error noise.

Stderr does not travel down a pipe by default

find / -name "*.py" | grep src

Only the found filenames (stdout) flow into grep. The flood of “Permission denied” errors go straight to your terminal on stderr. To pipe everything — results and errors alike:

find / -name "*.py" 2>&1 | grep src

A realistic multi-step example

You want to audit which Python processes are consuming the most memory, save the top 5 to a file, and discard any errors:

ps aux 2>/dev/null | grep '[p]ython' | sort -k4 -rn | head -5 > python-top.txt

shreyash  1234  2.1 12.3 python train.py
shreyash  5678  1.8  9.1 python serve.py
...

Breaking it down:

• ps aux 2>/dev/null — list processes, discard any errors
• grep '[p]ython' — keep Python lines (the bracket trick prevents grep itself from matching)
• sort -k4 -rn — sort by column 4 (memory %) in reverse numeric order
• head -5 — keep the top 5 lines
• > python-top.txt — save the result, overwriting any previous file

Quick reference

In one breath

Every process gets three streams: stdin (0, input), stdout (1, normal output), stderr (2, errors) — and stdout and stderr both reach your screen but are separate channels. The pipe | wires one command’s stdout into the next’s stdin, so ps aux | grep python | wc -l composes three tools into one answer with no temp files. Redirection sends a stream to or from a file: > overwrites (destructive — it erases the target the instant the shell opens it), >> appends, < feeds a file into stdin. Errors need their own handling because stderr does not travel down a pipe: 2> sends it to a file, 2>&1 (or &>) merges it into stdout, and 2>/dev/null throws it away.

Practice

Before the quiz, assemble one pipeline and reason about its streams. From access.log, you want the count of lines containing 500, with any read errors discarded and the number appended to daily-counts.txt. Write the single line. Then answer: in make build > out.log, why do the compiler’s error messages still show up on your screen, and what one addition captures them into the log too?

A question to carry forward

You can now wire tools together with | and capture the results with >. But step back and ask: what is actually flowing through those pipes? So far, mostly grep to filter and wc to count — useful, but the pipe’s real magic shows only when each stage transforms the stream. Picture a raw access log and the question “which ten IP addresses hit us most?” Answering it needs a little assembly line: slice out the IP column, sort the lines, collapse identical neighbours into counts, sort again by that count, take the top ten. Each of those is a tiny single-purpose tool — cut, sort, uniq, wc — and strung together with the pipes you just learned, they turn a four-million-line log into a ten-line leaderboard. Those building blocks are the next lesson, and they close out this chapter.