If you have multiple consecutive spaces in the string, use the last solution, the for loop. You can use any of the following solutions if you have a multiline string.

Using tr Command

Use the tr command to replace space with newline in bash.

text="This is a sample string."
new_text=$(echo "$text" | tr ' ' '\n')
echo "$new_text"

This
is
a
sample
string.

We initialized the text variable with a sample string. Next, we piped the output of the echo command to the tr command, which translated (replaced) space (' ') with newline (\n).

We captured the output of the echo "$text" | tr ' ' '\n' command using substitution syntax ($(...)) and assigned to the new_text variable. Finally, we used the echo command to print the new_text variable’s value on the bash console.

Using awk Command

Use the awk command to replace space with newline in bash.

text="This is a sample string."
new_text=$(echo "$text" | awk -v RS=" " '{print}')
echo "$new_text"

This
is
a
sample
string.

This code snippet is similar to the previous example; however, we piped the echo command output to the awk command this time. The awk is used for text processing. It operates per line by default, but we can customize it to work with fields and records (lines) using special actions and patterns.

In the above example, we used the awk command to replace space with a newline. How? For that, we used the -v option to set the record separator variable (RS) to a space character (" "). By default, the awk takes the newline character as the value of the RS variable.

So, modifying the RS to a space character meant that each space-separated word would become a separate record for processing. For each record, the print command was executed to print the current record, followed by a new line. This effectively replaced spaces with newlines in the output.

Alternatively, we can use the following solution:

text="This is a sample string."
new_text=$(echo "$text" | awk '{gsub(/ /, "\n"); print}')
echo "$new_text"

This
is
a
sample
string.

Here, we used the gsub, an AWK function to globally substitute all occurrences of a space character (/ /) in the text with a newline character ("\n").

Using sed Command

Use the sed command to replace space with newline in bash.

text="This is a sample string."
new_text=$(echo "$text" | sed 's/ /\n/g')
echo "$new_text"

This
is
a
sample
string.

We used the sed command, which is used to process and transform text, typically used for performing text substitution, deletion, insertion and more.

In the above example, we used the sed command to substitute text. For that, we wrote the sed command sequence enclosed in single quotes; for instance, 's/ /\n/g'. This sed sequence instructed to perform a substitution operation on the input text.

The sequence of the sed command was comprised of various components; let’s learn them below:

• s (stands for substitute) – It told the sed to find and replace.

• / (delimiter character) – It separated the different parts of the sed command: pattern to search for, replacement, and any flags.

• space (between first and second /): This was the pattern to search for. In our case, it was a single-space character.

• \n (newline) – It was the replacement part; when the pattern (space) was found in the input, it would be replaced with a newline.

• g (stands for global) – This flag told sed to perform the replacement globally, meaning all occurrences of the pattern in a line would be replaced with the newline character, not just the first occurrence.

This effectively split the input text into separate lines wherever there was a space character.

Using Parameter Expansion

Use parameter expansion to replace space with newline in bash.

text="This is a sample string."
new_text="{text// /$'\n'}"
echo "$new_text"

This
is
a
sample
string.

This example is similar to the previous ones, but we used parameter expansion this time. The parameter expansion ({text// /$'\n'}) took the value of the text variable, found all occurrences of a space character, and replaced them with a newline character. How? Following is a brief explanation of each component:

• ${...} was the basic syntax of parameter expansion; here, ... meant anything that you want to accomplish; for instance, text// /$'\n' in our case.

• // was a pattern-matching operator used in parameter expansion. When used as ${parameter//pattern/string}, it matched all occurrences of the provided pattern in the parameter (in this case, the variable text) and replaced them with the given string.

• ' ' was the pattern we searched for in the text variable. It was a single-space character.

• $'\n' was an ANSI-C quoted string representing a newline character (\n). The $'...' syntax interpreted backslash escapes in bash, so $'\n' was replaced with an actual newline character.

This was how we replaced space with newline in the text.

Using for Loop

Use the for loop to replace space with newline in bash.

text="This is a sample string."
new_text=""
for current_word in $text; do
    new_text="$new_text$current_word\n"
done
echo -e "$new_text"

This
is
a
sample
string.

We used a for loop in the above example to iterate over the $text. In each iteration, we concatenated the $new_text, $current_word, and a newline (\n) as "$new_text$current_word\n" and assigned its value to the new_text variable.

Once the loop was over, we used the echo command with the -e option to print the value of the new_text variable on the bash console.

Note that the -e option was used to interpret backslash escapes in the given string (in this case, new_text), allowing us to include special characters with specific meanings; for example, \n means a new line.

Searching for strings in text files is a common task in bash, used in scenarios like log file analysis and configuration file searches. This article explores various methods to check if file contains String, including both case-sensitive and case-insensitive approaches.

2. Introduction to Problem Statement

Let’s consider a log file named server.log:

2023-11-24 10:00:00 INFO Starting server process
2023-11-24 10:00:05 ERROR Failed to bind to port 8080
2023-11-24 10:00:10 INFO Server listening on port 9090
2023-11-24 10:01:00 WARN Database connection timeout

Our goal is to check if file server.log contains string Error in it.

The expected output is something like this:

Error found in server.log.

3. Using grep

The grep is a command-line utility optimized for searching text data for lines matching a regular expression.

if grep -q "Error" server.log; then
  echo "Error found in server.log."
else
  echo "Error not found in server.log."
fi

Explanation:

• grep -q "Error" server.log: This command searches for string Error in server.log. Here, -q stands for "quiet". It causes grep to not output anything but to exit with status 0 if the pattern is found, and 1 otherwise.
• The if statement then checks the exit status of grep. If grep finds the string, the first block ("Error found in server.log.") is executed; if not, the else block executes.

Case-Insensitive Search:

By default, grep search is case-sensitive. To perform a case-insensitive search, use the -i flag: grep -iq "Error" server.log.

Performance:
grep is highly optimized for searching text, making it fast and efficient for this task.

4. Using awk

The awk is a versatile programming language designed for pattern scanning and processing.

Let’s use awk with if to achieve our goal:

if awk '/Error/ {found=1; exit} END {if (!found) exit 1}' server.log; then
  echo "Error found in server.log."
else
  echo "Error not found in server.log."
fi

Explanation:

• This awk command scans server.log for the pattern Error.
• When Error is found, awk sets a flag found to 1 and exits immediately.
• In the END block, awk exits with status 1 if the flag found is not set, indicating that the pattern was not found.

Case-Insensitive Search:

To make the search case-insensitive in awk, we can use the tolower function:

if awk 'tolower($0) ~ /Error/ {found=1; exit} END {if (!found) exit 1}' server.log; then
  echo "Error found in server.log."
else
  echo "Error not found in server.log."
fi

Performance:
The awk is powerful for text processing but might be slightly slower than grep for simple string searches. However, it offers more flexibility for complex data manipulations.

3. Using Bash Conditional Expressions

Bash conditional expressions are a powerful feature in shell scripting, allowing for decision-making based on the evaluation of conditions within a script.

Let’s use conditional expression to check if file contains string:

if [[ $(cat server.log) == *"Error"* ]]; then
  echo "Error found in server.log."
else
  echo "Error not found in server.log."
fi

Here, cat filename.txt outputs the content of the file server.log, and the conditional [[ ... == *"Error"* ]] checks if the content contains "Error".
Case-Insensitive Search:
Bash does not directly support case-insensitive matching in this context. However, you can convert the file content and the search string to the same case:

if [[ $(cat server.log | tr '[:upper:]' '[:lower:]') == *"error"* ]]; then
  echo "Error (case-insensitive) found in server.log."
else
  echo "Error (case-insensitive) not found in server.log."
fi

Performance:
For smaller files, this method is quick and efficient. However, for larger files, its performance can degrade due to the need to read the entire file content.

6. Using sed with grep Command

The sed (Stream Editor) is a powerful and versatile text processing tool that performs text transformations on an input stream (a file or input from a pipeline).

Here, we will use sed for preprocessing and grep for final check. Let’s see with the help of example:

if sed -n '/Error/p' server.log | grep -q .; then
  echo "Error found in server.log."
else
  echo "Error not found in server.log."
fi

Explanation:

• The sed command searches server.log for lines containing the string Error and prints them.
• The output of sed (the matched lines) is passed to grep.
• grep -q . checks if there is any output from sed. If there is at least one line (meaning at least one line in server.log contained "Error"), grep exits with a status of zero.
• If grep exits with a status of zero, indicating that sed found at least one line containing "Error", the condition in the if statement is considered true, and the commands following then are executed. If no lines are found, the condition is false, and the commands after then are not executed.

Let’s understand more about sed expression sed -n '/Error/p' server.log used in above command:

• sed: This is a stream editor for filtering and transforming text.
• -n: This option suppresses automatic printing of pattern space. It means sed will not print anything unless explicitly told to do so.
• '/Error/p': This is a sed command enclosed in single quotes. It tells sed to search for lines containing the string Error and print those lines (p stands for print). The /Error/ is a pattern that sed looks for in each line of the input.
  server.log: This is the file sed reads from. sed will process each line of this file, looking for the pattern Error.

7. Using Bash Loops

This method involves iterating over each line of a file to search for a string. This approach is very slow and should be only used while searching in smaller files.

found=0
while IFS= read -r line; do
  if [[ $line == *"Error"* ]]; then
    found=1
    break
  fi
done < server.log

if [[ $found -eq 1 ]]; then
  echo "Error found in server.log."
else
  echo "Error not found in server.log."
fi

Performance:
Bash loops are straightforward but can be slower, especially for larger files.

8. Searching for Multiple Strings in File

While working on script, there are often situations where we need to search based on multiples patterns rather than single pattern.

There are multiple ways to do it. Let’s see with the help of examples:

Using grep Command:

if grep -Eq 'Error|Warn' server.log; then
  echo "Error or warn found in server.log."
else
  echo "Error or warn not found in server.log."
fi

This command uses the pipe | as a logical OR to search for "Error" or "Warn".

Using awk Command

if awk '/Error/ || /Warning/' server.log; then
  echo "Error or Warning found in server.log."
else
  echo "Error or Warning not found in server.log."
fi

This awk command checks if either Error or Warn is present in server.log.

Using Bash Loops

found=0
while IFS= read -r line; do
  if [[ $line == *"Error"* ]] || [[ $line == *"Warning"* ]]; then
    found=1
    break
  fi
done < server.log

if [[ $found -eq 1 ]]; then
  echo "Error or Warning found in server.log."
else
  echo "Error or Warning not found in server.log."
fi

This Bash loop manually iterates through each line of server.log, checking for Error or Warn.

9. Searching for String in Multiple Files

To search across multiple files, use grep with a file pattern.

if grep -q "Error" *.log; then
  echo "Error found in log files."
else
  echo "Error not found in log files."
fi

We can also provide filenames separated by space.

if grep -q "Error" server1.log server2.log; then
  echo "Error found in log files."
else
  echo "Error not found in log files."
fi

10. Performance Comparison

It’s important to test how fast each method works so we can choose the best one.

We’ll create a big input server.log with 1 million lines, and test each solution on it to search pattern "Error" in the file.
To Benchmark their performance, here is the script:

#!/bin/bash

# Sample file and string
file="server.log"
string="Error"

# Function to measure using grep
measure_grep() {
    time if grep -q "$string" "$file"; then
        echo "grep: String found."
    else
        echo "grep: String not found."
    fi
}

# Function to measure using awk
measure_awk() {
    time if awk "/$string/ {found=1; exit} END {if (!found) exit 1}" "$file"; then
        echo "awk: String found."
    else
        echo "awk: String not found."
    fi
}

# Function to measure using sed
measure_sed() {
    time if sed -n "/$string/p" "$file" | grep -q .; then
        echo "sed: String found."
    else
        echo "sed: String not found."
    fi
}

# Function to measure using Bash loop
measure_bash_loop() {
    time {
        found=0
        while IFS= read -r line; do
            if [[ $line == *"$string"* ]]; then
                found=1
                break
            fi
        done < "$file"

        if [[ $found -eq 1 ]]; then
            echo "Bash loop: String found."
        else
            echo "Bash loop: String not found."
        fi
    }
}

# Function to measure using Bash Conditional Expressions
measure_bash_conditional() {
    time {
        if [[ $(cat "$file") == *"$string"* ]]; then
            echo "Bash conditional: String found."
        else
            echo "Bash conditional: String not found."
        fi
    }
}

# Execute the functions
echo "Measuring grep..."
measure_grep

echo "Measuring awk..."
measure_awk

echo "Measuring sed..."
measure_sed

echo "Measuring Bash loop..."
measure_bash_loop

echo "Measuring Bash Conditional Expressions..."
measure_bash_conditional

Measuring grep...
grep: String found.

real    0m0.073s
user    0m0.015s
sys     0m0.000s

Measuring awk...
awk: String found.

real    0m0.209s
user    0m0.125s
sys     0m0.015s
Measuring sed...
sed: String found.

real    0m0.252s
user    0m0.187s
sys     0m0.000s

Measuring Bash loop...

Bash loop: String found.

real    1m39.973s
user    0m50.156s
sys     0m34.875s
Measuring Bash Conditional Expressions...

Bash conditional: String found.

real    0m6.878s
user    0m0.687s
sys     0m2.796s

The grep command is fastest of all as it is meant for searching text data.

11. Conclusion

In this article, we have different ways for checking if file contains a String. Let’s highlight important points:

• For simple string searching tasks in a file, grep proves to be the most efficient tool in terms of speed and CPU usage.
• While awk and sed offer more versatility for complex text processing, they are less efficient for straightforward string searches. For example: Once it’s confirmed that the file includes the string, substitute ‘Error’ with ‘Exception’ and proceed with similar replacements.etc.
• Bash loops and conditional expressions are significantly slower and less efficient for this task, and their use should be limited to cases where command-line tools like grep, awk, or sed are not viable.

In this article, we will see how to check if output contains String in Bash using grep, Conditional Expressions, awk, sed commands with various options.

2. Introduction to Problem Statement

We will use ls -l to list directories and files in long format and search for .txt string in the output.
For example:

-rw-r--r-- 1 user user      0 Jul  6 09:47 Customers.xlsx
-rw-r--r-- 1 user user      0 Jul  5 03:50 File1.txt
-rw-r--r-- 1 user user      0 Jul  5 03:50 File2.txt
-rw-r--r-- 1 user user      0 Jul  5 03:50 File3.txt
-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

Our goal is to search for .txt in ls -l command’s output. Expected output is:

-rw-r--r-- 1 user user      0 Jul  5 03:50 File1.txt
-rw-r--r-- 1 user user      0 Jul  5 03:50 File2.txt
-rw-r--r-- 1 user user      0 Jul  5 03:50 File3.txt

3. Using grep Command

The grep is a text-searching utility optimized for pattern matching, making it highly efficient and fast for such tasks.

Let’s see with help of example:

if ls -l | grep -q ".txt"; then
  echo "Text files found."
else
  echo "No text files found."
fi

Explanation:

• This method uses grep -q ".txt", where -q stands for "quiet". It causes grep to not output anything but to exit with status 0 if the pattern is found, and 1 otherwise.
• The if statement then checks the exit status of grep. If grep finds the string, the first block (echo "Text files found.") is executed; if not, the else block executes.

3.1 Searching for Pattern Using grep

We can use grep with -E option to search for pattern in the Output.

Suppose we need to find all .txt files in a folder. In this scenario, we can utilize *.txt as the pattern to match these specific files.

if ls -l | grep -qE "*.txt"; then
  echo "Text files found."
else
  echo "No text files found."
fi

The -E option enables extended regular expressions, allowing us to use patterns like *.txt. This method is particularly effective for pattern matching within scripts.

3.2 Searching with case-insensitive string

By default, grep search is case-sensitive. To do a case-insensitive search, use -i option.

if ls -l | grep -iq ".TXT"; then
  echo "Text files found."
else
  echo "No text files found."
fi

4. Using Bash Conditional Expressions

Bash conditional expressions are a powerful feature in shell scripting, allowing for decision-making based on the evaluation of conditions within a script.

4.1 Searching for Pattern

In Bash, the `[[ ... =~ ... ]] construct is used for regular expression matching. Here, the =~ operator matches the left-hand side $output with the regular expression $pattern on the right-hand side.

Let’s see with the help of example:

if [[ $(ls -l) == *".txt"* ]]; then
  echo "Text files found."
else
  echo "No text files."
fi

Here, if the match is found, the if block will execute. The [[ $output =~ $pattern]] condition evaluates as true and the message "Text files found." is printed. Otherwise, the condition evaluates as false, and the else block will execute with the message "No text files.".

4.2 Searching with case-insensitive string

In Bash version 4 and later, we can use the shopt -s nocasematch option to enable case-insensitive matching for the [[ ]] conditional expressions.

Example with shopt:

shopt -s nocasematch
if [[ $(ls -l) == *"file.txt"* ]]; then
  echo "file.txt found (case-insensitive)."
else
  echo "file.txt not found (case-insensitive)."
fi
shopt -u nocasematch

In this example, shopt -s nocasematch enables case-insensitive pattern matching within [[ ]], and shopt -u nocasematch is used to unset this option after the check.

This method is generally fast, especially for simple patterns and smaller outputs. However, it may be less efficient for complex patterns or large outputs compared to grep.

5. Using awk

awk is a powerful scripting language for text processing and is typically used for data extraction.
Let’s use awk to search string .txt in the output.

ls -l | awk '/\.txt/ {found=1} END {if(found) print "Text files found."; else print "No text files."}'

Explanation:

• ls -l command to list file in the directory.
• | is used to pass output of ls -lto awk command.
• awk scans the output of ls -l for lines containing .txt.
• It sets a flag found when the pattern is matched.
• The END block in awk is used to print the final output based on the found flag.

This method is highly efficient, particularly for complex data manipulation.

6. Using sed with grep Command

The sed (Stream Editor) is a powerful and versatile text processing tool that performs text transformations on an input stream (a file or input from a pipeline).

Here, we will use sed for preprocessing and grep for final check. Let’s see with the help of example:

if ls -l | sed '/\.txt/!d' | grep -q .; then
  echo "Text files found."
else
  echo "No text files."
fi

Explanation:

• sed '/\.txt/!d' deletes all lines not containing .txt.
• The output is then piped to grep to check if there’s any remaining text.
• This method combines sed for preprocessing with grep for final detection.

Let’s understand more about sed '/\.txt/!d':

• sed: This is the stream editor command in Unix/Linux, used for parsing and transforming text.

• '/.txt/!d': This is the expression passed to sed, consisting of two parts:

  • /.txt/: This part is a pattern match. It looks for lines that contain the string .txt. The dot . is escaped as . to indicate a literal period character, since in regular expressions, a dot is a special character that matches any single character.

  • !: The exclamation mark is used here to negate the pattern match. It means "for lines that do not match the pattern".

  • d: This is the delete command in sed. It deletes the lines from the output.

While sed is efficient for text processing, this method may be less performant due to the use of multiple commands and pipes.

7. Conclusion

In Bash, checking if a command’s output contains a specific string can be accomplished through various methods, each with its advantages. grep stands out for its simplicity and efficiency in pattern searching. Bash conditional expressions offer a built-in solution without external dependencies. awk provides a robust approach for more complex text processing needs. Lastly, sed, combined with grep, offers a versatile approach, though it might be less efficient for simple tasks.

This is Line number 1.
This is Line Number 2.
It is Line number 3.
I am Line 4.

This is Line number 1.
This is Line Number 2.
It is Line number 3.
I am Line 4.

Greetings! Welcome to Java2Blog!

Using cmp Command

Use the cmp command to check if two files are the same in Bash.

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File2.txt"
if cmp -s "$file1" "$file2"; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are the same.

In the above example, we initialized the file1 and file2 variables with the two different text file locations. Then, we used the if statement with the cmp command to check if the content of $file1 and $file2 was the same.

If it was the same, then the echo from the if block was executed; otherwise, the else block would be executed.

How did this command work? The cmp command compared the $file1 and $file2 files byte by byte and helped to find whether the given two files were identical or not. Here, the -s option (alternatively used as --silent) suppressed the output of differences between $file1 and $file2.

If the files were identical, the cmp produced no results but if they differ, by default, the cmp returns the byte offset and the line number where the first difference occurred.

Let’s take another example where the files are different.

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File3.txt"
if cmp -s "$file1" "$file2"; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are different.

Using Hash Values

Use sha1sum to calculate hash values to check if two files are the same in Bash.

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File2.txt"
hash1=$(sha1sum "$file1" | cut -d ' ' -f 1)
hash2=$(sha1sum "$file2" | cut -d ' ' -f 1)
if [ "$hash1" == "$hash2" ]; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are the same.

Here, we used the sha1sum to generate a SHA1 hash for the $file1 and piped the output to the cut command to grab the hash only. Because the sha1sum returned the output as hash filepath.

With the cut command, we used the -d option to specify the delimiter for splitting the input into fields, while -f 1 was used to select the first field, the hash, which we stored in the hash1 variable.

We repeated the process for getting a hash for $file2 and stored it in the hash2 variable. Then, we used the if statement with the == operator to check if both hashes are the same. If so, then we ran the echo inside the if block to display a message saying both files are the same.

If you have an error saying something similar to the bash: /home/user/File1.txt: Permission denied then run the chmod u+x filename.txt command on the bash console to allow permissions.

Let’s take another example below:

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File3.txt"
hash1=$(sha1sum "$file1" | cut -d ' ' -f 1)
hash2=$(sha1sum "$file2" | cut -d ' ' -f 1)
if [ "$hash1" == "$hash2" ]; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are different.

Similarly, you can use md5sum and sha256sum to calculate hash values to check if two files are the same in Bash.

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File2.txt"
hash1=$(md5sum "$file1" | cut -d ' ' -f 1)
hash2=$(md5sum "$file2" | cut -d ' ' -f 1)
if [ "$hash1" == "$hash2" ]; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are the same.

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File2.txt"
hash1=$(sha256sum "$file1" | cut -d ' ' -f 1)
hash2=$(sha256sum "$file2" | cut -d ' ' -f 1)
if [ "$hash1" == "$hash2" ]; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are the same.

Using -ef Operator

Use the -ef operator to check if two files are the same in Bash.

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File2.txt"
if [ "$file1" -ef "$file2" ]; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are the same.

Let’s take another example below:

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File3.txt"
if [ "$file1" -ef "$file2" ]; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are different.

Using diff Command

Use the diff command to check if two files are the same in Bash.

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File2.txt"
diff "$file1" "$file2" > /dev/null
if [ $? -eq 0 ]; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are the same.

In the above example, we used diff command to compare the $file1 and $file2 line by line. We used the > operator to redirect the standard output of the diff command to the /dev/null device, which would effectively discard the output. In other words, this line performs the comparison silently without displaying any anything on the console.

Next, we used the if statement with the -eq operator to check the exit status of the previously run diff command. If the exit status was equal to 0, it meant the specified files were the same; otherwise, different.

In Bash, the exit status of the previously executed command is stored in the $? variable. By convention, an 0 exit code indicates successful execution (no errors), while a non-zero represents some error that occurred during execution.

Let’s have a look at another example below:

#!/bin/bash
file1="/home/user/File1.txt"
file2="/home/user/File3.txt"
diff "$file1" "$file2" > /dev/null
if [ $? -eq 0 ]; then
    echo "Both files are the same."
else
    echo "Both files are different."
fi

Both files are different.

Use the diff command with the -q option to check if the two files are the same. For this solution, you do not have to use if-else statements. It will show nothing if the files will be the same.

#!/bin/bash
diff -q /home/user/File1.txt /home/user/File3.txt

Files /home/user/File1.txt and /home/user/File3.txt differ

Use the diff command with the -c option to retrieve a comparison of two files in context mode.

#!/bin/bash
diff -q /home/user/File1.txt /home/user/File3.txt

*** /home/user/File1.txt        2023-07-14 05:20:08.852722832 +0000
--- /home/user/File3.txt        2023-07-20 09:58:58.717339962 +0000
***************
*** 1,4 ****
! This is Line number 1.
! This is Line Number 2.
! It is Line number 3.
! I am Line 4.
\ No newline at end of file
--- 1 ----
! Greetings! Welcome to Java2Blog!
\ No newline at end of file

In the above output, the * character was used to display the File1.txt related things, while the - character was used to show File3 related things. In the first two lines of the output, we got file locations with date and time.

The *** 1,4 **** represented the range of lines in File1.txt. However, the --- 1 --- indicated the number of lines in the File3.txt. Remember, we would have a range of lines for File3.txt as well if it would have multiple lines.

How did we identify the differences in the output?

• The + meant the line was not present in the first file. Remove it from the second file or insert it in the first file to match.
• The - is similar to the + with a little difference. It means the line existed in the first file but not in the second file. Remove it from the first file or insert it in the second file to match.
• The ! represented that the line requires modifications to match.

Use the diff command with the -u option; it is similar to the context mode but without redundant details.

#!/bin/bash
diff -u /home/user/File1.txt /home/user/File3.txt

--- /home/user/File1.txt        2023-07-14 05:20:08.852722832 +0000
+++ /home/user/File3.txt        2023-07-20 09:58:58.717339962 +0000
@@ -1,4 +1 @@
-This is Line number 1.
-This is Line Number 2.
-It is Line number 3.
-I am Line 4.
\ No newline at end of file
+Greetings! Welcome to Java2Blog!
\ No newline at end of file

Let’s modify any letter’s case in any file. We modified the File1.txt by replacing the This with this in line 2. Now, if we run the diff command with the -q option, then it will say the files differ; see the following example.

#!/bin/bash
diff -q /home/user/File1.txt /home/user/File2.txt

Files /home/user/File1.txt and /home/user/File2.txt differ

What happened here? By default, the diff command does case-sensitive comparison. As we modified the letter’s case, the files became different.

Use the diff command with the -i option for case-insensitive comparison.

#!/bin/bash
diff -i /home/user/File1.txt /home/user/File2.txt

Now, we got nothing because the File1.txt and File2.txt were found identical.

Can we have any solution which can open an editor for us to modify the files if there is some difference that we don’t need? Yes, see the following section.

If you are looking for a standard output in coloured form, then replace the diff with colordiff as colordiff FirstFilePath SecondFilePath. Don’t forget to install colordiff using the sudo apt install colordiff command. Like diff, it will only show the output if the specified files are different; otherwise, not.

Using vimdiff Command

Use the vimdiff command to compare two files in Bash. You must install Vim Editor using sudo apt install vim to use this solution.

#!/bin/bash
vimdiff /home/user/File1.txt /home/user/File3.txt

OUTPUT:

As you can see in the above screenshot, both files were opened side by side and the differences were highlighted.

Use :wq and hit Enter to exit from the file.

Using cksum Command

Use the cksum command to check if the two files are the same.

cksum /home/user/*.txt

3232966274 79 /home/user/File1.txt
3232966274 79 /home/user/File2.txt
1972114799 66 /home/user/File3.txt

The cksum command is used to produce a CRC (Cyclic Redundancy Check) checksum for all the files in the given directory and output the checksum values with respective file names/paths and byte count. It would grab the files from the current directory if any path was not specified.

We used the cksum to get checksum values for all the .txt files in the specified directory. If the checksum values are the same, the files are the same; otherwise, not.

Do we have a solution to remove the duplicate files if found? Can we see the history before deleting them? Yes, let’s go to the following section.

Using rdfind Command

Use the rdfind command with the -deleteduplicates option to find duplicate files in the given directory and subdirectories, and delete them.

rdfind -deleteduplicates true /home/user

Now scanning "/home/user", found 3 files
Now have 3 files in total.
Removed 0 files due to nonunique device and inode
Total size is 193 bytes or 193 B
Removed 1 files due to unique sizes from list. 2 files left.
Now eliminating candidates based on first bytes:removed 0 files from list. 2 files left.
Now eliminating candidates based on last bytes:removed 0 files from list. 2 files left.
Now eliminating candidates based on sha1 checksum:removed 0 files from list. 2 files left.
It seems like you have two files that are not unique
Totally, 80 B can be reduced.
Now, making results file results.txt
Now deleting duplicates:
Deleted 1 files.

Using the -dryrun option before using -deleteduplicates helps in previewing the results rather than deleting them immediately.

That’s all about Check if two files are same in Bash.

In this article, we will see how to get last line of output in Bash using tail, awk, sed, mapfile, and head commands with various options.

2. Introduction to Problem Statement

We will use ls -l to list directories and files in long format and get the last line of output using various commands.

For example:

-rw-r--r-- 1 user user      0 Jul  6 09:47 Customers.xlsx
-rw-r--r-- 1 user user      0 Jul  5 03:50 File1.txt
-rw-r--r-- 1 user user      0 Jul  5 03:50 File2.txt
-rw-r--r-- 1 user user      0 Jul  5 03:50 File3.txt
-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

Our goal is to get last last line of the above output. Expected output is:

-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

3. Using tail Command

tail command in the unix system is used to get the last lines of files. By default, it shows the last 10 lines of its input to standard output. We can use tail -n 1 option to get last line of output.

ls -l | tail -n 1

-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

We use the ls- l command to list directories and files from the current directory in the long format and pipe its output to the tail command.

Here, tail -n 1 gives output as the last line because -n parameter is used to specify the number of lines to be extracted from output. Since we want to get only last line, we can use 1 with -n option.

We can capture output to variable using $(..) syntax.

OUTTUT = $(ls -l | tail -n 1)

Alternatively, we can also use process substitution as shown below:

tail -n 1 <(ls -l)

-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

Here, we used the process substitution (<(...)), which allowed the command’s output to be treated as the file. In our case, the command was the ls -l.

Using the process substitution syntax, we passed the output of the ls -l command as an input to the tail -n 1 command, which extracted the last line from it and displayed it on the Bash console.

Let’s say we want to get the last line of output, but only for .txt files. We can pipe grep command with -E option with pattern to the output of ls -l command and pipe it to tail -n 1 command.

ls -l | grep -E '\.txt$' | tail -n 1)

-rw-r--r-- 1 user user      0 Jul  5 03:50 File3.txt

Grep command can be used as an output to ls -l in all below methods, and we won’t repeat it for the rest of the article.

4. Using awk Command

awk is a powerful scripting language for text processing and is typically used for data extraction.

Let’s use awk to get last line of the output.

ls -l | awk 'END{print}')

-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

Let’s understand more about awk ‘END{print}’.

In the above example, the 'END{print}' represented the awk program, which consisted of the awk pattern-action pair.

Inside awk, END block is a special pattern that triggers action after all input lines have been processed.print without argument is action which instructs awk to print last line of the output.

5. Using sed Command

sed (Stream Editor) is a powerful and versatile text processing tool that performs text transformations on an input stream (a file or input from a pipeline).

To get the last line of output, we can use the $ address, which represents the last line of the output and use it with p command.

ls -l | sed -n '$p'

-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

While using the sed -n '$p', the -n instructed the sed to suppress printing all lines automatically. In contrast, the $p consists of an address ($, representing the last line) and action (p, printing the matched line).

6. Using mapfile Command

The mapfile command can be used to read entire output of ls -l into array and print the last element of the array.

ls -l | mapfile -t arr; echo "${arr[-1]}"

-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

Here:

• ls -l generates the file list.
• The output is piped to mapfile, which stores it in an array named arr.
• ${arr[-1]} is used to get the last element of the array, which is the last line of the ls -l output.

7. Using head Command

head command is command in Unix-like systems that is typically used to display the beginning of a text file or input.

Here, idea is to reverse the output of ls -l using tac command and get the first line from tac output using head -n 1.

ls -l| tac | head -n 1

-rw-r--r-- 1 user user      0 Jul  5 09:26 TestFile.md

Let’s understand more about the commands.

The tac command is the reverse of the cat command in Bash. It read all lines of input (whether from a file or a standard input; in our case, it was from the previous process in the pipeline) and wrote them line by line to the standard output in the reversed order.

head -n 1 gives the first line of the output as we have reversed ls -l output using tac command; it represents the last line of the ls -l command.

8. Performance

tail -n 1 is the fastest of all the approaches as it is optimized to read from end of the stream, and it should be used in this scenario.

awk and sed options are slower than tail command for large datasets as they process each line and hold the last line in memory, only printing it at the end. This means it reads the entire output, which can be less efficient with very large datasets. It is still good to learn these methods if we want to perform any operation on last line.
For example: Replace word "abc" with "xyz".

mapfile option is less efficient as it reads the output to the array, which can be memory intensive for large datasets.

9. Conclusion

In this article, we explored various ways to get last line of the output. We also compared the performance to choose the best method. Understanding these methods can help us to use method based on our use case and requirements.

We can use an if statement with different variations to determine whether the provided Boolean variable is true. Note that the examples covered in this section will be similar. We will learn the first example in Use if with true Condition in detail, but later, we will discuss the section which will differ from the previous example(s). Let’s begin with the first example below.

Use if with true Condition

Use the if statement with the true condition to check if boolean is true in Bash.

bool_var=true
if $bool_var; then
   echo "The Boolean is true."
else
   echo "The Boolean is false."
fi

The Boolean is true.

First, we have a variable bool_var, which was assigned the value true. The data type of this variable was Boolean, meaning the variable’s value can be either true or false. Note that we don’t have to specify the variable type in Bash explicitly.

Next, we encountered an if statement. The if statement was used to decide based on the provided condition. In the above example case, we used the condition, $bool_var, to check the value of the bool_var variable.

When we utilised $bool_var within the condition to expand the value of the bool_var variable, which was true in this case, essentially, it became equivalent to writing if true; then. Now, let’s break down the if statement further.

The if keyword indicated the start of the if statement, whereas the $bool_var was the condition being evaluated. Since it expanded to true, the condition was considered true. Next, we used the then keyword to indicate the start of the if block, which needs to be executed if the condition is true.

Inside the if block, we had an echo command to print the corresponding message to let the user know if the value of the provided Boolean variable was true to false. Using the else block is optional, but better to show a relevant message if the specified condition did not meet.

In the above code fence, the else indicated the start of the else block to be executed if the specified condition was false. We also had an echo statement in the else block to display a message. Finally, the fi keyword was used to mark the if statement’s end in Bash.

Let’s take another example where the bool_var was set to false.

bool_var=true
if $bool_var; then
   echo "The Boolean is true."
else
   echo "The Boolean is false."
fi

The Boolean is false.

Use if with [[ ]] Command

Use the if statement with the [[ ]] command to check if boolean is true in Bash.

bool_var=true
if [[ $bool_var == true ]]; then
  echo "The Boolean is true."
else
  echo "The Boolean is false."
fi

The Boolean is true.

In the above example, we used the double square brackets ([[ ... ]]) that indicated a conditional expression in Bash, and it allowed for more advanced comparisons and pattern matching. In this case, we checked if the value of bool_var was equal to the true. To check equality, we used the == operator.

Let’s experiment with the above code by modifying the value of the bool_var variable and setting it to false. See the following example.

bool_var=false
if [[ $bool_var == true ]]; then
  echo "The Boolean is true."
else
  echo "The Boolean is false."
fi

The Boolean is false.

Remember, using the -eq operator for comparison in the above example is not recommended, as it is used to compare numeric values. We will learn it in a while.

Until now, we have learned two ways (the true condition and the [[...]] command), and both are case-sensitive, which means if you set the bool_var to True as bool_var=True, it will not execute the if block. Instead, the else block will be executed (if it is specified); otherwise, you will get nothing.

Can we still find if the Boolean is true by avoiding this case-sensitivity thing? Yes. Let’s learn how to do it.

Use if with (( )) Command

Use the if statement with the (( )) command to check if Boolean variable is true in Bash.

bool_var=1
if (( $bool_var )); then
  echo "The Boolean is true."
else
  echo "The Boolean is false."
fi

The Boolean is true.

First, we initialised the bool_var variable with 1; this variable denoted a Boolean value, where 0 typically denoted false and 1 indicated true. Next, we encounter an if statement. The if statement allowed us to conditionally execute a block of code based on a specified condition.

For the condition part, we used (( ... )), which was an arithmetic expansion construct in Bash. It allowed us to perform mathematical operations and evaluations. In our case, we were using it to evaluate the value of the bool_var variable.

Since the value of bool_var was 1, the expression became (( 1 )). In arithmetic evaluation, any non-zero value is considered true, while zero is considered false. Since the value of bool_var was 1, the expression (( 1 )) was evaluated to true. As a result, the echo statement inside the if block was executed to print a message saying the Boolean is true.

Let’s have another example setting the bool_var to 0.

bool_var=0
if (( $bool_var )); then
  echo "The Boolean is true."
else
  echo "The Boolean is false."
fi

The Boolean is false.

Use if with -eq Operator

Use the if statement with the -eq operator to check if Boolean is true in Bash.

bool_var=1
if [[ $bool_var -eq 1 ]]; then
  echo "The Boolean is true."
else
  echo "The Boolean is false."
fi

The Boolean is true

We have already learned the [[ ]] command and the meaning of 1 while working with Boolean-type variables. Here, we used the -eq operator to evaluate the expression inside the [[ ]]. If the expression was evaluated to true, the if block would be executed; otherwise, the else block; remember that the -eq operator is used when we are supposed to do a numeric comparison.

Using case Statement

Use the case statement to check if the specified Boolean is true in Bash.

bool_var=true
case $bool_var in
  true)
    echo "The Boolean is true."
    ;;
  false)
    echo "The Boolean is false."
    ;;
esac

The Boolean is true

The case and if-else statements are used interchangeably. You can continue with any of them based on the project requirements. The above code performed a conditional check based on the value of the bool_var variable, which was set to true.

Then, we used the case statement to set up a conditional block where different patterns were matched against the $bool_var variable’s value. Those patterns are given below:

• true) – Represented the first pattern, checked in the case statement. It matched the value of $bool_var when it was equal to true.
• false) – Denoted the second pattern that was being validated in the case statement. It matched the $bool_var variable’s value when it was equal to false.

We used an echo command for both patterns to display the corresponding message on the Bash console. The ;; was used to indicate the case block’s end; they also separated the different patterns and their corresponding code blocks from one another. Finally, we used the esac keyword to mark the end of the case, closing the conditional block.

At this point, you must have a question popping into your mind why do you need to use a case statement when the goal can be achieved via if-else statements? Before going into the details, see the following example and predict its output.

mobile_missing=false
if [ $mobile_missing ]
then
    echo "Lost Mobile."
else
    echo "Not Lost Mobile."
fi

Lost Mobile.

We expected the else block to be executed in the above code fence because we set the mobile_missing variable to false. But it did not work as it was supposed to be. Why?

In Bash, variables don’t have types, so there is no such thing as a Boolean variable or value like true or false. In fact, all bash variables are just strings. When we test a variable/string in bash without specifying the type of test (-n or -z), it will default to a -n (non-zero length string) test.

The -n determines if the specified string operand size is non-zero; if so, it returns true. In contrast, the -z checks if the mentioned string operand size is zero; it returns true if it is zero length.

So [ $mobile_missing ] is equivalent to [ -n $mobile_missing ]. As long as $mobile_missing contains at least one character, it was evaluated as true.

The expression was always true since false (just a string) contained five characters. It doesn’t matter what string we set mobile_missing to (true, false, 0, 1); the expression would always be true because mobile_missing was the non-zero length.

This is where the case statement came into the picture. Let’s transform the above example into a case as follows:

mobile_missing=false
case $mobile_missing in
  true)
    echo "Lost Mobile."
    ;;
  false)
    echo "Not Lost Mobile."
    ;;
  *)
    echo "Invalid Input."
    ;;
esac

Not Lost Mobile.

Let’s have another example where the missing_mobile was set to an empty string.

mobile_missing=""
case $mobile_missing in
  true)
    echo "Lost Mobile."
    ;;
  false)
    echo "Not Lost Mobile."
    ;;
  *)
    echo "Invalid Input."
    ;;
esac

Invalid Input.

In the above example, the default case (*)) was used, which was matched if any of the above patterns did not match.

Until now, we learned how to check a single Boolean variable to find if that is true; what if we are required to validate multiple variables?

Validate Multiple Boolean Variables in Bash

Use the case statement to validate multiple Boolean variables and check if it is true.

bool_var1=true
bool_var2=false
case "$bool_var1$bool_var2" in
  "00" | "falsefalse")
    echo "Both variables are false."
    ;;
  "01" | "falsetrue")
    echo "'bool_var1' is false and 'bool_var2' is true."
    ;;
  "10" | "truefalse")
    echo "'bool_var1' is true and 'bool_var2' is false."
    ;;
  "11" | "truetrue")
    echo "Both variables are true."
    ;;
  *)
    echo "Invalid input."
    ;;
esac

'bool_var1' is true and 'bool_var2' is false.

After defining two Boolean variables, we used the case statement to check the concatenation of bool_var1 and bool_var2 using "$bool_var1$bool_var2" expression. We used this expression to concatenate the values of bool_var1 and bool_var2 into a single string for comparison.

After that, the case statement checked the value of the concatenated string against different patterns and executed the corresponding code block for the matching pattern. Let’s analyse the different patterns below:

1. "00" | "falsefalse") – This pattern matched when both variables were set to false.
2. "01" | "falsetrue") – It matched when bool_var1 was false and bool_var2 was true.
3. "10" | "truefalse") – It matched when bool_var1 was true and bool_var2 was false.
4. "11" | "truetrue") – This pattern matched when both variables were set to true.
5. *) (default pattern): This pattern was used as a catch-all for any other value that did not match the previous patterns.

We used the echo command to display a corresponding message for all the patterns.

We covered both input possibilities: a string (true/false) and numbered form (1/0). Whereas the | represented the logical OR operator.

Let’s specify the numbered values; see the following example:

bool_var1=0
bool_var2=1
case "$bool_var1$bool_var2" in
  "00" | "falsefalse")
    echo "Both variables are false."
    ;;
  "01" | "falsetrue")
    echo "'bool_var1' is false and 'bool_var2' is true."
    ;;
  "10" | "truefalse")
    echo "'bool_var1' is true and 'bool_var2' is false."
    ;;
  "11" | "truetrue")
    echo "Both variables are true."
    ;;
  *)
    echo "Invalid input."
    ;;
esac

'bool_var1' is false and 'bool_var2' is true.

So, we learned various ways to check whether Boolean is true. We also checked which Boolean variable is true if we had multiple variables. You can go ahead with any way that suits your project requirements better.

That’s all about Bash check if boolean is True.

Use the ping command in bash to check if the host is reachable or not.

#!/bin/bash
host="example.com"
ping -c 1 $host > /dev/null
if [ $? -eq 0 ]; then
  echo "Host is reachable."
else
  echo "Host is not reachable."
fi

Host is reachable.

In this bash script, the ping command is used to check if the host "example.com" is reachable or not. In bash, the ping command is used to send ICMP Echo Request packets to the given host and wait for the ICMP Echo Reply. Here, the -c option is used to tell the ping command to send only 1 packet. We can change the value of -c based on the number of packet requests we want to send. Then, > /dev/null; redirected the output to discard it.

The $? is a special variable in bash. It holds the exit status code of the last executed command. In this case, it will capture the exit code of the ping command if it is equal to 0 it will display the message "Host is reachable." on the console and if the exit code is non-zero the message "Host is not reachable." will be displayed. On the execution, of the given command we can see the host example.com is reachable.

Have a look at the below example to see the response of the ping command if the host we are trying to connect is not available.

#!/bin/bash
host="my.com"
ping -c 1 $host > /dev/null
if [ $? -eq 0 ]; then
  echo "Host is reachable."
else
  echo "Host is not reachable."
fi

Host is not reachable

In the above example, the ping command is used to connect host my.com which is not available on the internet. As a result, the connection failed and the ping command returned the non-zero exit status to an if statement and the message Host is not reachable was displayed on the screen to indicate the connection failure.

Using curl Command:

Use the curl command in bash to check if the host is reachable or not.

#!/bin/bash
host="https://www.google.com/"
if curl --head --silent --fail $host >/dev/null; then
  echo "Host is reachable."
else
  echo "Host is not reachable."
fi

Host is reachable.

In this bash script, the curl command is used to send a HEAD request to check if the specified host "https://www.google.com/" is reachable. The --head option here tells the curl command to send only HTTP HEAD request to the host and the --silent option is used to suppress the progress output. And the --fail option instructed the curl command to send a non-zero status code in the case of request failure to show that the host is not reachable.

On execution of the given script, the message Host is reachable. is displayed on the screen to show that the connection is successfully established.

Note: Replace the "https://www.google.com/" with your desired host in the host variable.

Using nmap Command

Use the nmap command in bash to check if the host is reachable or not.

#!/bin/bash
host="example.com"
port=80

nmap_output=$(nmap -p $port $host)

if echo "$nmap_output" | grep -q "open"; then
  echo "Host is reachable on port $port."
  echo "Services available on the network:"
  echo "$nmap_output"
else
  echo "Host is not reachable on port $port."
fi

Host is reachable on port 80.
Services available on the network:
Starting Nmap 7.94 ( https://nmap.org ) at 2023-07-18 12:12 Pakistan Standard Time
Nmap scan report for example.com (93.184.216.34)
Host is up (0.22s latency).
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.96 seconds

In this example, the nmap is used to scan the network and check the availability of the given host "example.com" on specified port 80 including the services available on the network. Here, the -p is used to specify the port that we want to scan. The output of the nmap command is then stored in the nmap_output variable.

After that conditional statement is used to check if the nmap_output variable contains an open string indicating that the port is open. In the if condition first the output of echo "$nmap_output" is redirected to the grep command using the pipe | operator as input. Then, grep searched for the string open in the received input. Here, -q is used to suppress the output to only return the exit status.

If the given string is found the exit status code 0 (zero) will be returned and the if block will be executed. On the other hand, if the given string is not found non-zero exit status will be returned and the else block will be executed. On execution of the above code, we can observe that the host "example.com" is reachable at port 80 along with the services available at the network.

Note: Do not forget to install nmap if it is not already installed on your system to avoid errors.

To install nmap run the following command if you are using Ubuntu-based systems/Debian:

sudo apt-get update
sudo apt-get install nmap

And if you are using Windows Operating System download the nmap installer from the official nmap website and follow the given instructions for a successful installation. After installation run the above nmap code to verify if the nmap is installed or not.

Using ncat Command

Use the ncat command in bash to check if the host is reachable or not.

#!/bin/bash
host="google.com"
port=443
if echo -n | ncat --wait 1 $host $port; then
  echo "Host is reachable on port $port."
else
  echo "Host is not reachable on port $port."
fi

Host is reachable on port 443.

In this bash script, the ncat command is used to check if the host "google.com" is reachable at the given port 443 or not. In bash, the ncat is used to establish the network’s connections and manipulation. The echo -n is used to generate an empty output without a newline character which is passed to the ncat command. Then, ncat attempts to establish a connection with the host at the given port.

Note: The purpose of using echo -n before ncat is to initiate the connection without sending any data, just to test the reachability of the host and port.

Here, the --wait option is used to set the timeout of 1 second for the connection attempt. If the connection is established within the given time ncat will send the exit status 0 to indicate a successful connection. On the other hand, if the connection is not established at the given time ncat will terminate and send the non-zero exit code to indicate connection failure.

In the above case, we can observe that the host "google.com" is reachable at port 443.

That’s all about Bash check if host is reachable.

Use substitution syntax to capture the output of a single command into a variable in Bash.

date_variable=$(date)
echo "Today's date: $date_variable"

Today's date: Thu Jun  8 04:08:47 UTC 2023

We used the date command to get the current date and time. The date command was enclosed within the substitution syntax represented by the $(...), which we used to capture the output of the date command. After that, we stored the captured output in a variable named date_variable. Finally, we used an echo command to display the value of the date_variable with a custom string on the Bash console.

Use substitution syntax to capture the output of a pipeline into a variable in Bash.

file_count=$(ls | wc -l)
echo "$file_count"

7

In the above example, we used substitution syntax ($(...)) to grab the pipeline output. The pipeline included two commands: ls and wc -l.

The ls listed all files and directories in the current directory; this command generated the list of directories and file names as output. Note that the substitution syntax will not capture the output of the ls command, but it will capture the final result produced by the pipeline.

So, the output of the ls command was passed to the wc -l command using a pipe (|). The wc command counts the received input’s lines, words, and characters. In the above code fence, we used the -l option to count the lines in the received input.

In the pipeline, the previous command’s output is the next command’s input.

Now, the substitution syntax ($(...)) captured the output of the ls | wc -l pipeline and stored it in the file_count variable, which we further used with an echo command to display its value on the Bash console.

Use substitution syntax to capture the output of a multiline command to a variable in Bash.

string_value=$(echo "This is line one \
This is line two. \
This is line three.\
This is line four.")
echo "$string_value"

This is line one This is line two. This is line three.This is line four.

Again, we used the $(...) to capture the command output. It is similar to the first example, but the important point is writing a multiline command by enclosing them in the substitution syntax. To write the multiline command, you only need to write \ before moving to a new line (see the above example).

Use substitution syntax to capture the output of a for loop into a variable in Bash.

for_loop_output=$(for i in {1..5}; do \
    echo "This is Iteration: $i"; \
    echo "Now Processing File $i.txt"; \
done)
echo "$for_loop_output"

This is Iteration: 1
Now Processing File 1.txt
This is Iteration: 2
Now Processing File 2.txt
This is Iteration: 3
Now Processing File 3.txt
This is Iteration: 4
Now Processing File 4.txt
This is Iteration: 5
Now Processing File 5.txt

This example resembles the previous code snippet. As the for loop is a multiline process, we wrote a backslash (\) before moving to the new line. Then, we enclosed the entire for loop into the substitution syntax to capture its output and stored it into a variable called for_loop_output. Finally, we used an echo command to print its value on the Bash console.

Ensure you don’t have a space before and after the equal sign.

Using Backticks

Use backticks to capture the output of a single command into a variable in Bash.

date_variable=`date`
echo "Today's date: $date_variable"

Today's date: Thu Jun  8 04:46:06 UTC 2023

Use backticks to capture the output of a pipeline into a variable in Bash.

file_count=`ls | wc -l`
echo "$file_count"

7

Use backticks to capture the output of a multiline command to a variable in Bash.

string_value=`echo "This is line one \
This is line two. \
This is line three.\
This is line four."`
echo "$string_value"

This is line one This is line two. This is line three.This is line four.

Use backticks to capture the output of a for loop into a variable in Bash.

for_loop_output=`for i in {1..5}; do \
    echo "This is Iteration: $i"; \
    echo "Now Processing File $i.txt"; \
done`
echo "$for_loop_output"

This is Iteration: 1
Now Processing File 1.txt
This is Iteration: 2
Now Processing File 2.txt
This is Iteration: 3
Now Processing File 3.txt
This is Iteration: 4
Now Processing File 4.txt
This is Iteration: 5
Now Processing File 5.txt

All the code snippets are the same as we learned while using substitution syntax. The only difference is that we enclosed the commands within the backticks rather than the substitution syntax; see the above examples carefully.

Ensure you don’t have a space before and after the equal sign while using backticks to capture the command’s output.

Using read Command

Use the read command to capture the output of a single command into a variable in Bash.

date | read date_variable
echo "Today's date: $date_variable"

Today's date: Thu Jun  8 04:46:06 UTC 2023

Use the read command to capture the output of a pipeline into a variable in Bash.

ls | wc -l | read file_count
echo "$file_count"

7

Use the read command to capture the output of a multiline command to a variable in Bash.

echo "This is line one \
This is line two. \
This is line three.\
This is line four." | read string_value
echo "$string_value"

This is line one This is line two. This is line three.This is line four.

Use the read command to capture the output of a for loop into a variable in Bash.

for i in {1..5}; do \
    echo "This is Iteration: $i"; \
    echo "Now Processing File $i.txt"; \
done | read for_loop_output
echo "$for_loop_output"

This is Iteration: 1
Now Processing File 1.txt
This is Iteration: 2
Now Processing File 2.txt
This is Iteration: 3
Now Processing File 3.txt
This is Iteration: 4
Now Processing File 4.txt
This is Iteration: 5
Now Processing File 5.txt

The code snippets used in this section are the same as we used in the last two sections while using substitution syntax and backticks. This time, we did not enclose the commands within the backticks or substitution syntax, whether single-line or multiline commands, but we piped the output of these commands to the read command.

The read command reads the input received from the previous process and stores it in the specified variable. See the examples above to clarify the concept.

You must understand how pipelines work to use the read command.

That’s all about how to set output of command to variable in Bash.

Use the printf statement to convert the array to a comma-separated string in Bash.

array=("Jan" "Feb" "Mar" "Apr")
printf -v joined '%s,' "${array[@]}"
echo "${joined%,}"

Jan,Feb,Mar,Apr

We initialized an array containing string elements. Then, we used the printf statement to convert the specified array to a comma-separated string. How? The printf statement formatted and stored the elements of the array into the joined variable, which we specified using the -v parameter.

In the printf statement, the "%s" was the format specifier for strings, which told that every element of the array must be printed as a string value and followed by a comma. The "${array[@]}" expanded the array data into individual elements, which would be formatted via a printf statement.

So, the printf -v joined '%s,' "${array[@]}" line essentially concatenated the items of the array into a string separating every item with a comma. Finally, we used the echo command to print the value of the joined variable with the last comma (trailing comma) removed.

In the echo command, the "${joined%,}" expanded the value of the joined variable and removed the trailing comma using the ${parameter%,} syntax; here, the parameter was the joined variable. Here, the % character was used as a pattern to remove the trailing comma.

Alternatively, if you don’t want to use the -v parameter, you can store the comma-separated string into a variable as follows.

array=("Jan" "Feb" "Mar" "Apr")
array=$(printf '%s,' "${array[@]}")
echo "${array%,}"

Jan,Feb,Mar,Apr

This example is similar to the previous one, but we used syntax substitution syntax here. First, we used the substitution syntax represented by the $(...) to capture the output of the printf statement, which was further assigned to the array variable; finally, we displayed it on the Bash console using the echo command.

In both examples, the delimiter, a comma (,) in our case, is preceded by the %s.

Using awk Command

Use the awk command to convert an array to a comma-separated string in Bash.

array=("Jan" "Feb" "Mar" "Apr")
awk -v sep=',' 'BEGIN{
    ORS=OFS="";
    for(i=1;i<ARGC;i++){
        print ARGV[i],ARGC-i-1?sep:""
    }
}' "${array[@]}"

Jan,Feb,Mar,Apr

After initializing the array variable, we used the awk command to convert the given array into a comma-separated string. Let’s break down the use of the awk command to understand it.

We used the -v parameter to set the value of the sep variable; we used a comma as a delimiter and initialized the sep variable with a comma. Then, we used the awk script enclosed within the single quotes, which was run before processing the input. Following is a brief explanation of each component within the script.

• The BEGIN is an awk pattern, specifying the actions/operations to be performed before input processing. Here, the ORS=OFS="" set the ORS (Output Record Separator) and OFS (Output Field Separator) to the empty string, ensuring that no extra characters would be added between the items while printing them on the console.
• The for loop was used to iterate over the given arguments to the awk command. The print statement within the loop printed the current argument (ARGV[i]) followed by the specified separator (sep). However, it printed an empty string rather than the separator for the last element using the ternary operator (ARGC-i-1?sep:"").
• We used the "${array[@]}" to expand the array into separate items, which were passed as arguments to the awk command. These were the arguments on which the awk script iterated and printed them, followed by the given separator.

This way, the array was converted to comma-separated strings; you can see the output above.

Using IFS Shell Variable

Use IFS (Internal Field Separator), a shell variable, to convert the specified array to a comma separated strings in Bash.

array=("Mon" "Tue" "Wed")
IFS=,
output="${array[*]}"
echo "$output"

Mon,Tue,Wed

First, we declared and initialized the array containing three string elements. Then, we set the value of IFS, a particular shell variable, to a comma; this value would be used to separate the strings.

After that, we used parameter expansion to expand the array into separate items and joined them with the IFS separator, which we explicitly specified. Finally, we assigned the comma-separated strings to the output variable, which was further used with the echo command to display the results on the console.

Alternatively, we can do as follows:

array=("Mon" "Tue" "Wed")
(IFS=,; echo "${array[*]}")

Mon,Tue,Wed

This example is comparable to the last one. Here, in the second line of code, we created a subshell represented by (...), which was executed to temporarily modify the value of the IFS variable and printed the comma-separated string using the echo command. Note that the semi-colon was used to separate the commands from one another.

Why did we create a subshell to mutate the IFS value and expand the array elements? Because we want to ensure that the changes would be temporary and not affect the rest of the script. Using this approach, you can convert the array into a comma-separated string by temporarily mutating the value of the IFS variable and expanding the array elements using parameter expansion ("${array[*]}").

What to do to split the array into a comma-separated string without mutating the IFS variable? In that use case, we can use the following approach, which will only work as expected if the IFS variable is not mutated and set to its default value, a space character.

array=("Mon" "Tue" "Wed")
expandedArrayStr="${array[*]}"
echo "${expandedArrayStr//${IFS:0:1}/,}"

Mon,Tue,Wed

Again, we created an array named array having three string-type elements. Then, we used parameter expansion syntax to expand the array into a string and stored it in the expandedArrayStr variable. Finally, we used the echo command to print the comma-separated string on the Bash console. Let’s understand how a string was transformed into a comma-separated string.

In the echo "${expandedArrayStr//${IFS:0:1}/,}"command:

• The ${IFS:0:1} expression expanded to the first character of the IFS variable, which by default contained a whitespace character, including a tab, newline, and a space. This extracted a substring from the IFS beginning at index 0 with a length of 1. Since the IFS was a single character, so we retrieved the first character of IFS effectively.
• The ${expandedArrayStr//${IFS:0:1}/,} expression was a parameter expansion with a pattern substitution (whose syntax was ${inputString//pattern/replacement}). This expression substituted all occurrences of the given pattern (${IFS:0:1}, the first character of IFS) in the expandedArrayStr with replacement (,).
  Using this, all occurrences of the IFS‘s first character in expandedArrayStr would be replaced with a comma, effectively converting the array to a comma-separated string.
• Finally, the echo command printed the results on the Bash console.

You must convert the array to a string to use this approach.

Using for Loop

To transform an array to a comma-separated string in Bash:

• Declare and initialize an array containing string items.
• Use the for loop to iterate over an expanded array. The array is expanded using parameter expansion.
• In each iteration of for, we used the += operator to join the current element followed by a comma.
• Use the echo command with ${parameter%,} syntax to remove the trailing command and print the values on the bash console.

array=("grapes" "guava" "orange")
output=""
for element in "${array[@]}"
do
    output+="$element,"
done
echo "${output%,}"

grapes,guava,orange

Here we used a for loop to join all elements.

That’s all about how to convert Array to Comma Separated String in Bash.