Terminals were the original method of interacting with programs: You type in input, programs respond with output. Over time, terminals implemented many additional features: limited foreground and background color support, modifying the cursor position, erasing content, and more. Where have these features gone when the pixel-based displays we know and love today came along?
Nowhere!
Today you have programs that can simulate (or emulate) these old terminals and draw the result in a window on your screen, hence terminal emulators. Along with that came surprisingly even more features. The XTerm terminal emulator is one of the base pieces of software in an X11 Linux system. Of course there came others, like Konsole for the KDE desktop, and some even implemented whole new protocols like Kitty, which allows displaying actual images in the terminal and downloading files from remote servers over just the terminal connection.
You can use these terminal features manually, but they are quite cumbersome to use, and there are libraries like ncurses to abstract them away. One of the more simple ones you can even include in your shell scripts easily:
echo -e '\e[31merror'
Prints “error” in red, useful for printing error messages that can be easily distinguishable from normal output. The -e parameter for echo is so it interprets the escape sequence \e as the ASCII ESC (escape) character, which all escape sequences have to start with. The ANSI control sequences then follow with an opening square bracket, then parameters delimited by semicolons and then a character for the command name. In this case the command is “Select Graphic Rendition” and the only parameter is 31, which sets the foreground color to red. With 0 you can reset the color and other attributes if needed.
A list of the graphics commands can be found here.
With \e[2K you can clear the current line and reset the cursor to the beginning with \e[G, to display progress indicators like the curl progress bar for example.
\e[3J is an addition by XTerm, which deletes the entire screen and the scrollback buffer.
Shell scripts are files with shell commands you can run, so you don’t need to type all the commands every time you want to use it.
Simple scripts just run the same commands each time you run them, but you can make scripts also accept parameters, read files, get input from the user and change the behavior according to that.
Shells have some specifics for these things, and this post assumes you use bash.
Let’s start with a script without any kind of input.
Maybe you want an inspirational quote each morning when you start your terminal.
A good start is to run and chain the commands in the terminal before writing them into a script, so you can run and modify them directly.
I found a website that gives you a daily quote, let’s download it with curl:
curl 'https://www.brainyquote.com/quote_of_the_day'
That spits out a lot of HTML.
Parsing HTML with the shell is a bit tricky, but can work for simple patterns you want to find.
I like to use GNU grep’s perl regular expression support for that, because GNU grep is installed practically everywhere.
You can use
grep -P -o -z -e '(?s)qotd-wrapper-cntr.*?<a.*?>.*?>\n*\K(.*?)(?=<.*?<\/a>)'
to extract all the quotes from the website. If you want you can use a website like regex101 that explains the regular expression, but you don’t need to do that.
With head -n 1 we can then extract the first quote.
You can chain the commands with pipes to get the first quote directly from the website.
If you put
curl 'https://www.brainyquote.com/quote_of_the_day' | grep -P -o -z -e '(?s)qotd-wrapper-cntr.*?<a.*?>.*?>\n*\K(.*?)(?=<.*?<\/a>)' | head -n 1
in your ~/.bashrc, you get a quote every time you open a terminal.
Getting a quote every time you open the terminal may be a bit much, so let’s see how we can limit it to one per day.
date +%D gets you the date. We just need a file to store the last quote date, compare it with the current one and only output the quote when the dates differ. We also need to write the current date back into the file.
To do that, we need to know about command substitution and if:
With command substitution, you can use the output of a command to construct a command.
It’s especially useful to assign the output of a command to a variable.
It works like $(command), but you should use "$(command)" again to get correct handling of spaces.
The command then gets executed before the rest of the line, end you can even nest them: "$(cat $(cat filenames.txt))".
An if-block is used to execute commands only when a condition is true. The format is:
if program; then
command_when_true
else
command_when_true
fi
You can leave out the else-part if you don’t need it.
The exist status of the program after the if determines which path is taken.
An exit code of 0 means the program completed successfully, and the true path is taken.
Anything else is interpreted as false.
To make these conditions easier, there’s the shell builtin (or external program) test or [ ] that provides conditions you can use. You can use man test to read all of them. Some important ones:
[ -z "string" ]
Tests if the string is empty.
[ -n "string" ] or just [ "string" ]
Tests if the string is not empty.
[ "string1" = "string2" ]
Tests if string1 and string2 are equal.
[ "string1" != "string2" ]
Tests if string1 and string2 are not equal.
[ $? -ne 0 ]
-ne compares numbers for inequality, and $? is a special shell variable that is the return code of the last command.
Use this to check if the last command failed.
You can use variables in the conditions the same like everywhere else.
Now we can test if the date is the same:
lastquote="$(cat ~/lastquote.date 2>/dev/null)" # don't print the error message if the file doesn't exist
currentDate="$(date +%D)" # Get the date
if [ "$lastquote" != "$currentDate" ]; then # check if the last print was today
# Print the quote
curl 'https://www.brainyquote.com/quote_of_the_day' | grep -P -o -z -e '(?s)qotd-wrapper-cntr.*?<a.*?>.*?>\n*\K(.*?)(?=<.*?<\/a>)' | head -n 1
echo "$currentDate" > ~/lastquote.date # write the date to the file
fi
Sometimes you want to execute commands repeatedly without having to type them multiple times.
There are 2 loops for this:
While is like if, but the commands get run again as long as the condition is true.
# Loops endlessly and prints "While!"
while true; do
echo "While!"
done
With for you can do something for every entry in a list. Lists in bash are just lists of words, where words are separated with spaces. You can combine this with another shell feature (globbing) to generate file lists in directories with files that match a pattern.
# Prints the names of all .txt files in the current directory
for file in *.txt; do # Using just * matches all names that don't start with a '.'
echo "$file"
done
All parameters are available in the special variables $x where x is a number > 1.
They are present in the order they are typed, which can make flags like seen in other programs difficult.
# use GNU getopt to parse options
params="$(getopt -l 'long-varaiant,option-with-argument:' -o 'of:' -- "$@")"
if [ $? -ne 0 ]; then
# getopt detected an error, so we exit. getopt should have already printed an error message
exit 1
fi
# Set the parameters to the ones ordered and validated by getopt
eval set -- "$params"
# clean up the variable if you want
unset params
# Loop over the parameters
while true; do
# case can compare a string with patterns or other string
case "$1" in
('-o'|'--long-varaiant')
# do something
shift # shift the parameters to the left. $2 is now $1...
;;
('-f'|'--option-with-argument')
# do something
shift 2 # shift the parameters by 2, since $2 belongs to this option, too.
;;
('--')
shift
break # Break out of the loop, -- is the end of the options
# * is the fallback pattern for everything
(*)
# Error, should not be able to happen
exit 1
esac # you have to end the cse block like this
done
# process the non-option parameters left in $1...
Sometimes you need to count things in a script. You can use arithmetic inside double brackets (()). If you need the result you can use $(()).
i=10
i=$((i + 1)) # $ is not needed inside (()) to use variables
((i += 1)) # you can also assign in (())
You can then use -eq and -ne in conditions to test numbers for equality and inequality, -lt and -gt for less than and greater than and -le and -ge for less or equal and greater or equal.
With this alone you should be able to make many functional scripts to automate things. Here are some things you can look up if you need them:
Environment variables are simple pairs of a name and a value that are passed to each command you run.
You can use the command env to print all current ones out.
You can define new ones with export NAME=VALUE.
You can also define exported variables for one command only: A=aaaa env will also list the variable A.
Shell variables are like environment variables, but are not passed to commands you run.
You can set them with NAME=VALUE.
To use a shell or environment variable, use $NAME, e.g. echo $HOME to print the home directory of the user.
echo is another “command” that’s usually not a separate program, but build into the shell for things like printing variables.
When you want to use 2 variables directly after each other, you have to also enclose them in {} so the shell knows it’s not just one name: echo ${HOME}${USER}
Now that you know basic commands and how to read their help test, it is time to re-visit the command syntax.
A shell command is build like this:
[ENVIRONMENT_VARIABLES] command [parameter] [parameter] ...
The parts in [] are optional, and … means you can add as many parameters as you like (there is technically a limit, but you won’t reach that when manually typing commands).
The spaces are significant: Parameters, the command and the environment variable definitions are all separated by at least one space. When you want something to include spaces, surround it in "".
One handy thing is that variables are replaced before anything about the command gets run:
LS="ls -l"
$LS ..
This will run ls -l .., because $LS gets evaluated first.
But what if you want to print literally $HOME?
You can use '' which suppresses variable replacement: echo '$HOME'.
When using variables where you don’t know if the value contains spaces, always encase them in "", or you might get nasty surprises:
file="my super cool filename with spaces.txt"
cat $file
This will try to print the contents of the files my, super, cool, filename, with, spaces.txt, not my super cool filename with spaces.txt.
Sometimes you want the parts of the variable value to be treated as separate though:
ls_options="-a -l"
ls "$ls_options"
This would make ls complain about an invalid option.
To understand redirection, we have to make a small detour over file descriptors:
Whenever a program opens a file, it gets a positive number back that references this file in the operating system for the program. That number is a file descriptor.
The first 3 file descriptors have a standard meaning:
Programs can close file descriptors to make the number available for a new file and duplicate a file descriptor to another number, so they reference the same actual file.
The shell exposes this in the form of I/O redirection.
Let’s say a program writes to the standard output, but you only want the error messages.
You could redirect the standard output away if you want to keep it.
Closing it would maybe make the program crash, as it expects to be able to write the standard output (it’s standard, yknow?).
For this there are 2 special files:
/dev/null: Acts as an infinite sink for data. The operating system just discards any data written to this file./dev/zero: An infinite source of 0 bytes. Writes are also discarded.You can use FD<NAME to open the file descriptor FD for reading (piping data into FD) from the file NAME and FD>NAME to make writes go to the file NAME (piping from FD to the file NAME).
When writing, the file gets overwritten. If you want to append the data instead, use FD>>NAME.
If FD is not given, 0 is assumed for < and 1 for > and >>.
You can also specify another file descriptor on the right side, but you have to use & so the shell knows it’s not a filename: FD1<&FD2.
With that knowledge we can now pipe away the pesky output into oblivion!
ls >/dev/null will print nothing, which is exactly what we were after in this case.
Another case is where you don’t want the error information and just the normal output: COMMAND 2>/dev/null will discard the standard error for COMMAND.
And lastly, you may not want any output from a program:
COMMAND >/dev/null 2>/dev/null: Works, but is much to type.COMMAND >/dev/null 2>&1: First redirects standard output to /dev/null and then standard error to standard output, which we set to /dev/null.COMMAND 2>&1 >/dev/null: This outputs the standard error to the terminal, as we’re first piping standard error top standard output and then standard output to /dev/null. The shell applies the redirection in the order you give them.Linux terminal programs tend to specialize in one thing or area, but complex tasks are still possible by combining them.
The shell can transfer the standard output of one program into the standard input of the next program with |.
Let’s say you have a copy of Mobby-Dick in the file mobby.txt and you want to find all lines where “Mobby” is used and send them to a friend. Sadly, Emails don’t support such large attachments.
You could do
grep Mobby mobby.txt > found.txt
gzip found.txt
or combine it onto one line:
grep Mobby mobby.txt | gzip > found.gz
If your friend wants to send you back the lines that also contain “whale”, they could do
gunzip -c found.gz | grep whale | gzip > found_whale.gz
and send you back the file without clobbering the directory with temporary uncompressed files.
These are the main handy features for interactive shell use. The next post is about creating shell scripts.
]]>The main difference is that the environment Android apps are run in is quite different from a standard Linux environment.
Here are the main differences:
A normal Linux system has directories under the filesystem root: /bin, /etc, /lib, /tmp and /home are the most important ones.
Android apps get no access to these directories, so Termux replicates this layout under it’s own files, /data/data/com.termux/files/usr.
The home directory is placed at /data/data/com.termux/files/home instead.
Regular Linux programs expect the folders to be there under /, and don’t work under Termux without configuring/patching them to use the new path.
A Linux system has a user for each, well, user.
Android has a separate user for each app, and each “user” can only access that app’s files.
Typically Android apps also don’t have access to the root user, which can be enabled by rooting your device.
Most Android devices are phones, which have processors designed by ARM with the arm64 instruction set.
Desktop computers have mostly x86(_64) processors by Intel or AMD.
That means even if the program you want to run doesn’t need the standard linux file layout, your processor won’t even know how to execute it.
For privacy reasons, apps cannot see other apps that are running on the system.
That also translates to programs like ps in Termux, which can only list the programs running in Termux.
Other parts of /proc are also blocked from apps for this reason.
Newer Android versions also block some system calls for security, you might see bad syscall errors in Termux.
The display on a Linux system is managed by the X server or the Wayland compositor.
Android has its own system for managing the display called SurfaceFlinger.
This is incompatible with Linux programs, which is why you have to use an X server and view it with a VNC app, use Termux:X11 or other solutions to use graphical programs in Termux.
If you don’t know any terms, check out the glossary ↩
lsechocatcpmvmkdirrmdirrmtouchfind to search in files and foldersgrep to search for text with regular expressionscurl or wgetFor every line you can write commands in the shell (that is, not for every line that another program outputs), a prompt with some information will be displayed.
Shells typically display the user you are logged in as and the path to the directory you are currently in.
When you see ~ in the current path it means the user’s home directory.
To change the directory you can use the cd “command” (quotes, because it’s not an actual program you can run, but rather provided by the shell itself).
In contrast to windows, subdirectories and files are separated with / from the parent directory, not \.
To change the directory, use cd PATH:
cd ~
cd linux
cd ..
cd linux/blog
cd ../..
A new line in these code snippets means you have to press enter. Pressing enter lets the shell know you finished your command and the shell can now run it.
This snippet first navigates to your home directory, then to the directory named “linux”, then back out again.
.. is a special directory that means “the parent directory”, so you can easily navigate to parent directories.
Then it navigates to the blog subdirectories of the linux directory. That demonstrates that you have to use /to separate a folder name from the sub-folder name.
The last command navigates back to your home directory. It navigates to the parent directory of the parent directory, so 2 levels up, which is the home directory in this case.
When your names include spaces, you have to enclose them with "": cd "my path with spaces".
The shell would interpret each word separated by the spaces as its own path otherwise.
If your path includes " or $, you have to use '' instead.
Shells also support something called tab completion, so you don’t have to type as much. When you enter a command or path and hit enter, the shell looks for command names or files/directories that match what you typed. If there is only 1 result to this search, the shell will complete the command or path for you.
The above snippet likely doesn’t work for you, as you may not have the directory ~/linux/blog.
To create a directory, use mkdir PATH, or mkdir -p PATH if your path contains a parent directory that doesn’t exist.
To delete an empty directory, use rmdir PATH.
To create an empty file, use touch PATH.
To delete a file, use rm PATH.
If you want to delete a folder with contents, use rm -r PATH (make sure you want to delete everything in it though).
The -r in rm -r PATH is an option. Options are optional information you can pass to programs. There are 2 types of options:
- and the option letter or --LONGER_OPTION_NAME for options with more than one lettercurl -o google.html https://www.google.com would download https://www.google.com to the file google.html.To copy use cp SOURCE DESTINATION.
To move, use mv instead.
But how do you know which files and folders you can copy and move?
You can use ls PATH to list them.
For the current directory ls is enough.
You can print files to the terminal with cat FILE, or edit them with an editor of your choice.
Personally I use nano because displays the keys you need and I’m too lazy to learn a full terminal text editor.
To download files, you can use curl or wget with the URL you want to download:
curl -o google.html https://www.google.com
If you want do do something in the general theme of one of the applications I showed, chances are it can actually do it.
Most applications support the flags -h or --help which shows more information about the application.
For more in-depth information there is the separate command man, which you may have to install yourself.
Use it with man COMMAND_NAME.
That was an introduction into some basic commands. The next post is about useful shell features.
Here is my next post in this series.
If you don’t know any terms, check out the glossary ↩