Redirection and piping are two useful features in bash scripting that sysadmins and developers use often. In this guide, we will discuss what is Bash Redirection and how to work with Redirection in Bash with example commands.
Before understanding how redirection works, it is important to learn what file descriptor is.
Table of Contents
What is a File descriptor?
When you open a file in Linux, each file will be assigned with an Integer and this information is stored in the kernel. This way the kernel knows what files are opened and which process has opened the files. The assigned Integer number is what we call file descriptor (shortly FD).
By default, every program will start with three file descriptors.
- FD 0 -> Standard Input(stdin) -> Keyboard
- FD 1 -> Standard Output(Stdout) -> Display(Terminal)
- FD 2 -> Standard Error(Stderr) -> Display(Terminal)
You can see the file descriptors in /dev
directory:
$ ls -l /dev/std* lrwxrwxrwx 1 root root 15 Aug 27 21:32 /dev/stderr -> /proc/self/fd/2 lrwxrwxrwx 1 root root 15 Aug 27 21:32 /dev/stdin -> /proc/self/fd/0 lrwxrwxrwx 1 root root 15 Aug 27 21:32 /dev/stdout -> /proc/self/fd/1
Stdin (FD0) gets input from the keyboard. stdout (FD1) and stderr (FD2) are sent to the terminal. When using Pipes and Redirections, you can change the way where input is passed from and output and errors are sent to.
Redirecting output to a file
As stated earlier, output (stdout) and error (stderr) of any program is sent to the terminal. You can use the redirection operator ">” to write the stdout and stderr to a file.
Take a look at the below example. I am running the uname -mrs
command and redirecting the output to a file named uname.log
.
$ uname -mrs > uname.log
$ cat uname.log
Linux 5.10.0-8-amd64 x86_64
Heads Up: When using ">"
operator, if the file is not available, it will be created. And if the file exists already, it will be overridden with new contents.
You can also use the file descriptor number for stdout(1) before the redirection operator to redirect output to a file.
$ uname -mrs 1> uname.log
Like I already mentioned, a single redirection operator (>
) will only override the contents of a file, if the file already exists. However, if you want to append the contents instead of overwriting to the same file, use double redirection operator (i.e. >>
). You can also use stdout file descriptor(1) here too.
$ whoami >> uname.log
$ echo $SHELL 1>> uname.log
$ cat uname.log Linux 5.10.0-8-amd64 x86_64 ostechnix /bin/bash
How to handle Stderr
Every program generates both output and error and both are sent to the terminal. In many cases, you may wish to either ignore the error or isolate the errors and redirect it to a separate file for troubleshooting.
Let's take a simple example of running the ls
command to see how this works. I am trying to list two files out of which only one is present. I am getting both my error and output in the terminal as I already said.
$ ls -l uname.log u1.log ls: cannot access 'u1.log': No such file or directory -rw-r--r-- 1 ostechnix ostechnix 48 Aug 27 07:37 uname.log
I am running the same ls
command again but this time redirecting the output to a file. As you can see from the output >
operator redirects stdout(1) to a file but stderr(2) is sent to the terminal.
$ ls -l uname.log u1.log > ls.op ls: cannot access 'u1.log': No such file or directory
To redirect stderr to a file use the "2>"
operator. It is mandatory to use the file descriptor for stderr(2) before the redirection operator > that will send the error alone to a file.
$ ls -l uname.log u1.log > ls.op 2> ls.err
$ cat ls.err ls: cannot access 'u1.log': No such file or directory
Now, stdout and stderr are written to separate files. You can also send stdout and stderr to a single file too.
$ ls -l uname.log u1.log 1> ls.op 2> ls.op
From bash 4.4, you can also use &>
sign to redirect both stdout and stderr to a file.
$ ls -l uname.log u1.log &> ls.op
What is /dev/null?
Null is a character special file that accepts input and discards the input and produces no output. To put this simply, null discards whatever you redirect to it.
$ ls -l /dev/null crw-rw-rw- 1 root root 1, 3 Aug 27 06:01 /dev/null
Why does the null matter in redirection? You might wonder. In some cases, you may wish not to print or store stdout or stderr. In that case, you can redirect either stdout or stderr to /dev/null which will discard the stream of input.
$ date > /dev/null $ date 1> /dev/null # Stdout to Null $ dateee 2> /dev/null # WRONG COMMAND, Stderr to Null $ date &> /dev/null # Stdout/Stderr to Null
Input Redirection in Bash
Similar to how you are redirecting the output and error to a file, you can also pass inputs to a command using input redirection operator (<
).
Let’s start with a simple word count program. Here, I am redirecting the contents of ostechnix.txt
file to the word count to find the number of lines.
$ wc -l < ostechnix.txt 10
You can also pass stdin file descriptor (0) when redirecting the input.
$ wc -l 0< ostechnix.txt 10
You can combine the input and output redirection operators as shown below.
$ wc -l < ostechnix.txt &> /tmp/wc.op
$ cat /tmp/wc.op 10
Input redirection will be used along with a while loop to read the content of a file line by line.
Take a look at the below example. I am passing the /etc/passwd
file as the input to the while
command. Here the read
command will read line by line and store it in the variable VAL
and further down in the loop condition is written to check if the user is available.
while read VAL; do NAME=$(echo $VAL | awk -F ":" '/1/ {print $1}' ) if [[ $NAME = "karthick" ]] then echo "User spotted" fi done < /etc/passwd
Replace "karthick" in the above code with your system username. Save the file and close it.
This is not an optimal way to achieve the task of finding the user but for demonstration purposes, this will do.
Now, make the script executable and run it:
$ chmod +x ostechnix.sh
$ ./ostechnix.sh
User spotted
Bash supports another form of input redirection known as Heredoc. For more details, refer the following guide:
Summary
In this guide, we learned a few important scripting concepts such as what is Bash redirection, what is file descriptor, how to redirect an output to a file, how to handle stderr, what is /dev/null and how it is used in Bash redirection, and finally we wound up the guide with Input redirection in Bash.
If you wish to learn Bash scripting, you should learn and get familiar with the concepts explained here. To make yourself comfortable, I suggest you to try all the commands that we have shown in this article in your system.
If you have any feedback or any tips that will help the Linux community and our readers to learn bash scripting, feel free to let us know in the comment section below.
Suggested read:
- Bash Scripting – Variables Explained With Examples
- Bash Scripting – Functions Explained With Examples
- Bash Scripting – For Loop Explained With Examples
- Bash Scripting – While And Until Loop Explained With Examples
- Bash Echo Command Explained With Examples In Linux
- The List Of Useful Bash Keyboard Shortcuts
- Difference Between Defining Bash Variables With And Without export
4 comments
So, you did not have the text file in the script but rather passed it in as an argument when you ran the script ?
For the purpose of demonstration, we have done it through terminal. For input redirection, the file is passed inside the script. In a real case, you can pass the file as an argument when invoking the script or hardcode the filename inside the script.
Very good explanation Karthick! Congrats.
Thanks for the feedback 🙂