IO Redirection and Pipes

For those of you reading the blog to catch up on a missed lesson, I wanted to mention that we started off the lesson with a review of all the concepts we learned up to now. If you have time, go back to previous blog posts and make sure you understand them. You don't have to memorise everything! You're not writing an exam, relax.

Also, I realised that I forgot to teach you about one of the most fun things about using Linux: using tiling window managers! If the lessons up till now have bored you, then wait till you see what a tiling window manager is and how it can change your workflow.

IO Redirection

IO stands for Input/Output. There are 2 concepts:

Output Redirection

The following command will simply output “Hello, World!”. You already know this.

echo Hello, World!

Now, if we add a right angle bracket (aka greater than symbol) and a filename, the command will write the text “Hello, World!” to that file.

echo Hello, World! > myFile

You can use the cat command to view the contents of myFile to verify.

This syntax, with the right angle bracket is called “Output Redirection” because it takes the output of the echo command and redirects it into a file. As you may guess, it works with any command.

ls > contentsOfDirectory

The above command will write the output of ls to the file.

Side note: you may notice that the contents of the file contentsOfDirectory after running that command are slightly different than what you would see if you ran the ls command normally. In particular, each file and directory would be written to the file on a new line. This is since ls gives different output depending on if the output is shown to you, or written to a file. (If you want to know why, ask me in person, or look it up). Note that this is not the case for all commands.

Now let's run the echo command again, shown below, but using the filename from the ls command.

echo Hello, World! > contentsOfDirectory

If you view the contents of that file, you will see “Hello, World!”. The previous contents would have dissappeared. This is because output redirection with ‘>’ overwrites the contents of the file. If you want to append to the file, use 2 right angle brackets.

echo Hello, World! >> contentsOfDirectory

After running that command, you will see 2 instances of “Hello, World!”, since the command appended to the file. Be careful when using output redirection with ‘>’ so that you don't accidentaly overwrite a file, because there is no ‘undo’ for this.

Input Redirection

Similarly, there is a way to use files as input to commands. This is called input redirection. For example, the following command will search for all occurences of ‘word’ in the file ‘myFile’:

grep word myFile

This command is taking myFile as an argument. Another way to do this is:

grep word < myFile

Now, the result will be the same, but we have used Input Redirection. The left angle bracket (<) is telling the computer to feed the contents of ‘myFile’ into the grep command as input. Notice that grep is no longer taking myFile as an argument, instead, myFile is part of the < syntax.

Input redirection is less commonly used, especially with basic commands, because most commands accept a file as an argument (like the first example of grep above), so input redirection is redundant. However, if you are programming, and your program expects input through standard input, you can do the following to feed it the contents of myFile as input:

./myProgram < myFile

You may find this useful if you are doing competitive programming (like for the CCC), and you want to test your program with some sample input. Doing it this way will eliminate the need to type in the input every time you want to test your program.

Pipes

Make a file named ‘myFile’ with the following contents:

This is a sentence.
Another sentence.
Here is one more.
Hello, World!

Now runnning the following command will search for the word ‘sentence’ in myFile:

grep sentence myFile

And running the sort command will output the alphabetically sorted contents of myFile.

sort myFile

Now, what if I want to find occurences of ‘sentence’, and then sort them? I could use output redirection like this:

grep sentence myFile > tmp

And then run the sort command on the file ‘tmp’.

sort tmp

But there's a better way, where you don't have to create a temporary file. It's like this:

grep sentence myFile | sort

The | character is usually found above the backslash on US keyboards. Here, it tells the computer to take the input from the first command (in this case, the grep command) and feed it directly to the second command as input. We say that the output of grep sentence myFile is ‘piped’ into sort. This syntax is known as a ‘pipe’ because you can think of the input of the 1st command as being sent through a pipe to the second command.

Pipes allow you to chain commands together. You can use multiple pipes in one line. For example:

echo "This is some more text" | cat myFile  - | sort

This command will first run the echo command, which will output “This is some more text”, and then the cat command will concatenate the contents of myFile and the input it receives through the pipe (the - tells it to take input through the pipe) and then the last command will sort the whole thing.