Pipes & Redirection: The Unix Philosophy in Action

Everything so far has treated commands as separate tools you run one at a time. This chapter introduces the idea that changes the game — the one that makes people fall in love with the command line. Unix tools were deliberately built to connect, passing their output straight into one another, so you can assemble a custom tool for the exact problem in front of you out of small, reliable parts. Once this clicks, you stop looking for the right command and start building it.

The Unix philosophy, in one sentence

There's a design idea underneath the whole Linux toolset, and it's worth stating plainly: write programs that each do one thing well, and make them speak a common language so they can be combined. That common language is plain text — lines of characters. Because ls, grep, sort, and wc all read and write plain text, the output of any one of them can become the input of any other.

This is completely unlike graphical apps, which are sealed boxes: a photo editor can't hand its result to a spreadsheet. Unix tools are the opposite — deliberately open on both ends, designed to be strung together. The pipe, which we'll reach shortly, is the connector. But first we need to see the three channels every program already has.

Every program has three streams

When a program runs, it comes wired with three standard streams — channels for text to flow in and out. You've been using them without noticing.

• stdin — standard input. Where the program reads its input from. By default, that's your keyboard.
• stdout — standard output. Where the program writes its normal results. By default, that's your screen.
• stderr — standard error. A separate channel just for error and diagnostic messages. Also goes to your screen by default, which is why errors and normal output appear mixed together.

The reason there are two output streams — one for results, one for errors — is subtle but brilliant: it means you can capture a program's real output to a file while still seeing its errors on screen, because they travel on different wires. Redirection and pipes are nothing more than rewiring these three streams to point somewhere other than the keyboard and screen. That's the whole trick.

Redirection: sending streams to files

The first way to rewire a stream is to point output at a file instead of the screen, using >:

$ ls -l > listing.txt

That runs ls -l but, instead of printing to the screen, writes its stdout into listing.txt. Your screen stays empty; the file holds the result. One catch worth burning in: > overwrites. If listing.txt already had contents, they're gone, replaced entirely.

When you want to add to a file rather than replace it, use >>:

$ echo "deploy finished at $(date)" >> deploy.log

>> appends to the end of the file, creating it if it doesn't exist and leaving any existing contents untouched. It's how log files grow over time — each event tacked onto the end.

You can redirect input too, with <, which feeds a file into a program's stdin:

$ sort < names.txt

And you can redirect the error stream separately using 2> (the 2 is stderr's stream number), which is how you keep error messages out of your results:

$ find / -name "*.conf" 2> /dev/null

That last line searches the entire system for .conf files and throws away the flood of "permission denied" errors — sending them to /dev/null, a special "black hole" file that discards anything written to it — so you see only the results that matter.

The pipe: the most important character in the shell

Redirection connects a program to a file. The pipe, written |, connects a program directly to another program — it takes the stdout of the command on its left and feeds it straight into the stdin of the command on its right, with no temporary file in between:

$ ls -l | grep ".log"

Read it left to right: ls -l produces a listing, and instead of hitting the screen, that listing flows through the pipe into grep, which keeps only the lines mentioning .log. The result — just the log files, in long format — lands on your screen. Two simple tools, combined into one you didn't have a moment ago.

And you can keep chaining:

$ cat access.log | grep "404" | wc -l

That reads the log, keeps only lines with "404" (not-found errors), and counts them (wc -l counts lines). In one line you've answered "how many 404s are in this log?" — a question with no dedicated command, assembled from three that each do one small thing.

The workhorse filters

Pipes are only as good as the tools you chain. A handful of small "filter" commands — programs that read text in and write transformed text out — do most of the heavy lifting:

• grep — keep only lines matching a pattern (from the last chapter, now supercharged by taking piped input).
• sort — put lines in order (sort -n for numeric order, sort -r to reverse).
• uniq — collapse adjacent duplicate lines into one; uniq -c counts how many of each.
• wc — count lines, words, or characters (wc -l for lines).
• head / tail — keep just the first or last N lines (head -20, tail -5).
• cut — pull out specific columns from each line.

Because uniq only removes adjacent duplicates, it's almost always paired with sort first. That pairing unlocks a genuinely useful recipe. In the standard web-server log format the visitor's IP is the first field of each line, so to find the ten busiest visitors:

$ cat access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn | head -10

Walk the pipeline: pull the IP from each line (cut), sort so identical IPs sit together, collapse-and-count them (uniq -c), sort those counts high-to-low (sort -rn), and keep the top ten (head). That's a real analytics query — the kind you might expect to need a dashboard for — expressed as five small tools in a row.

tee: watch and save at once

One more handy tool, because you'll want it: tee splits a stream in two, writing it to a file and passing it along the pipe so you can still see it or feed it onward:

$ ./run-tests.sh | tee results.log | grep "FAIL"

That runs your tests, saves the complete output to results.log for later, and simultaneously shows you just the failures on screen. Named after a T-shaped pipe fitting, it's the answer whenever you think "I want to keep this output and keep working with it."

Why this is the heart of the terminal

Step back and see what you've gained. In the last chapter, each command stood alone and did roughly what a file manager could. With redirection and pipes, the tools combine — and combinations are effectively infinite. You're no longer limited to the features someone decided to ship; you assemble the exact tool for the task at hand, right now, from parts that each do one small, dependable thing.

This is the answer to "why not just use a graphical app?" from chapter one, made concrete. A GUI gives you the buttons its designer imagined. The pipe gives you every combination of every tool on the system. That's not a small difference in convenience — it's a different kind of power, and it's why engineers live at the command line.

Next we'll turn from what commands do to who's allowed to do it: permissions, users, and the sudo command that governs who can touch what on a shared Linux system.