When working with Bash scripts, you may end up in a situation where you have to process a series of inputs using the same command. Fortunately, there is a way in Bash to achieve this in a more optimal way using HereDoc.
HereDoc, acronym for Here Document, is an input Redirection method to pass multiple inputs to a program or command. The concept of heredoc is not exclusively related to Bash alone. Many popular programming languages like Perl, Ruby, PHP support heredoc.
In this article, we will take a look at the syntax and usage of heredoc with some real-world use cases. All the examples in this article are created to be simple, so even a newbie can understand this concept easily. Let’s jump in and start playing with heredoc in Bash.
Table of Contents
HereDoc Syntax
The following diagram illustrates the syntax of heredoc.
Here,
- Command - Any command (cat, wc, mail, etc..) that accepts redirection.
- Redirection Operator (
<<
) - The default operator for HereDoc is<<
. It redirects the block of code to the command for processing. - Delimiter Token - The delimiter token denotes the start and end of the document (code block). The delimiter token can be anything but it should be identical. Typically you will see
EOF
is used as delimiter tokens which mean "End Of File Stream".
Printing MultiLine String Using HereDoc In Bash
Let’s start with a simple example of redirecting a multiline string and printing it to the terminal.
The cat
command accepts a stream of inputs and with heredoc, and you can redirect the block of code to print it to the terminal.
$ cat << EOF Something is wrong with the input file received for today. Contact the downstream team to get it corrected. ==> SLA MISSED <== EOF
Output:
Something is wrong with the input file received for today. Contact the downstream team to get it corrected. ==> SLA MISSED <==
Take a look at the above code snippet. I have three lines that are redirected to the cat
command. I am using EOF
as the delimiter. However, you can use anything as you wish but keep the starting and ending delimiter identical.
Let’s try with one more simple example. I am redirecting the same three-line to word count program. I am using a different delimiter (BLK
) here.
$ wc -l << BLK Something is wrong with the input file received for today. Contact the downstream team to get it corrected. ==> SLA MISSED <== BLK
Sample output:
3
Redirecting And Pipe In HereDoc
You can combine the output redirection operator with heredoc and redirect the output to a file instead of printing it to the terminal.
I am using the same example which I used in the previous section and redirecting the output to a file named log_op.txt
.
$ cat << EOF > /tmp/log_op.txt Something is wrong with the input file received for today. Contact the downstream team to get it corrected. ==> SLA MISSED <== EOF
The output of heredoc can be sent to the pipe operator for further processing.
$ cat << EOF | grep -i sla Something is wrong with the input file received for today. Contact the downstream team to get it corrected. ==> SLA MISSED <== EOF
Tab Suppression In HereDoc
When there are white spaces (tabs) in your code block and if you wish to suppress it, then use "-"
after the redirection operator. An important point to be noted is that only tabs will be suppressed, not spaces.
Take a look at the below example. I have added a conditional statement to the same example that we have seen in previous sections. The first two lines in heredoc are tabbed (4) and the third line is spaced (2).
if [[ $x = "err" ]] then cat <<- err_msg 1. Something is wrong with the input file received for today. 2. Contact the downstream team to get it corrected. 3. ==> SLA MISSED <== err_msg fi
When the code snippet is submitted, then my output will be as follows.
1. Something is wrong with the input file received for today. 2. Contact the downstream team to get it corrected. 3. ==> SLA MISSED <==
As you can see, line 1 and 2 tabs are suppressed, but in line 3 since spaces are used it is not suppressed.
Variable And Command Expansion In HereDoc
It is not that you can pass only strings within the heredoc code block. You can pass user-defined and environmental variables and run commands within the code block.
Take a look at the below example. Within the code block, I have one user-defined variable "${AUTHOR}"
, one environmental variable "${SHELL}"
, an external command "whoami"
.
When this snippet is submitted, variables and commands will be expanded then it will be redirected to the cat
command.
AUTHOR="OSTechNix" cat << EOF Author: ${AUTHOR} # USER DEFINED VARIABLE Article: Bash Heredoc I am using the ${SHELL} shell # ENV VARIABLE $(whoami) # EXTERNAL COMMAND EOF
Sample output:
Author: OsTechnix Article: Bash Heredoc I am using the /bin/bash shell karthick
You can enclose the starting delimiter with single quotes to suppress the expansion within the code block. This way everything within the code block will be treated as a string literal.
cat << 'EOF'
Author: ${AUTHOR} # USER DEFINED VARIABLE
Article: Bash Heredoc
I am using the ${SHELL} shell # ENV VARIABLE
$(whoami) # EXTERNAL COMMAND
EOF
Sample output:
Author: ${AUTHOR} Article: Bash Heredoc I am using the ${SHELL} shell $(whoami)
MultiLine Comments With HereDoc
As you may already know, Bash does not support multi-line comments. Using heredoc, you can create multi-line comments by redirecting the block of code to the no-op
command (:
).
The no-op
is bash built-in which takes the input and returns exit code zero. You can consider this as a synonym to the bash built-in "true" which also exits exit code zero.
: << 'COMMENTS' Author : OStechnix Article : Bash Heredoc BashV : 5.1.4 OS : PoP!_OS COMMENTS
Heads Up: Almost every text editor has the feature of selecting multiple lines and allows you to comment or uncomment using a keystroke. It is better to stick with this approach.
Escaping Special Characters In HereDoc
Code blocks may contain special characters. If you wish to escape the special characters, there are few ways to get it done.
You can enclose the delimiter with single or double quotes or prefix backslash with the delimiter. This way all the special characters will be escaped.
# SINGLE QUOTES ESCAPE
cat << 'EOF'
I am using the ${SHELL} shell # ENV VARIABLE
$(whoami) # EXTERNAL COMMAND
EOF
# DOUBLE QUOTES ESCAPE
cat << "EOF"
I am using the ${SHELL} shell # ENV VARIABLE
$(whoami) # EXTERNAL COMMAND
EOF
# BACKSLASH ESCAPE
cat << \EOF
I am using the ${SHELL} shell # ENV VARIABLE
$(whoami) # EXTERNAL COMMAND
EOF
Instead of escaping all the special characters, you can also escape particular special characters within the block by adding a backslash before any special character.
cat << EOF I am using the \${SHELL} shell # ENV VARIABLE $(whoami) # EXTERNAL COMMAND EOF
Sample output:
I am using the ${SHELL} shell karthick
HereDoc Use Case
Till now we have seen the core construct of heredoc and its basic usage. Now let’s see some real-life use cases. In my experience, I have used heredoc when working with ssh, and database clients where I have to run a group of commands.
Example 1: Running As Different User Inside The Script
In some cases, you may want to run certain parts of your code as a different user. In that case, you can use heredoc to redirect the commands to run as a different user.
Take a look at the below example. I am redirecting the block of code to the su
command which will switch the user to "ostechnix" and will create a file named test if it does not exist.
su - ostechnix << EOF if [[ ! -f /home/ostechnix/test ]];then touch /home/ostechnix/test echo "File Created" else echo "File exists" fi EOF
Example 2: Using HereDoc with DB Client
When you want to run a series of commands against a database, heredoc will come in handy.
Take a look at the below example. I am interacting with MongoDB client mongosh and within the heredoc, code block commands are passed to create a new database, collection, and adding a sample document.
mongosh << EOF use ostechnix db.data.insertOne({ "Site" : "OsTechnix", "DB" : "Mongo" }) db.data.find().pretty() EOF
You may or may not be knowing MongoDB but that is fine. This is just to show how to use heredoc to interact with db clients. You can use any db client like MySQL, psql, sqlite depending upon which database you are working with.
Example 3: Executing Remote Commands with HereDoc and SSH
When you wish to run commands over the remote server, then you can use heredoc in combination with ssh
command. Normally using ssh
command, you can run commands in the remote server in the following way.
$ ssh user@host "command"
You have to repeat the same command again and again if you wish to run more commands in the remote host. With heredoc, you can group all the commands and run them.
$ ssh -T user@host << EOF Command 1... Command 2.. ..... Command N.. EOF
When the same code needs to be executed across multiple nodes you can add for loop
along with heredoc. I am using the same file creation snippet which we have seen in the first example.
- The array variable "server" holds the list of server names.
- For loop iterates over the array variable.
- The file creating command is passed to the SSH command that will iterate over and create files in each server. Make sure you add
-T
flag to ssh command which will disable pseudo-terminal allocation.
declare -a server=( host1 host2 host3 ) for host in ${server[@]} do ssh -T user@${host} << EOF echo "Running at host - ${host}" if [[ ! -f /home/ostechnix/test ]];then touch /home/ostechnix/test echo "File Created" else echo "File exists" fi EOF done
Conclusion
Heredoc is an important concept to understand and use in Bash scripts. When you write a lot of scripts you will come to know more about heredoc and ways to optimally use it.
If you have never used heredoc before, start the terminal and try all the code snippets from the article to have a better understanding.