2013-01-24

Bash scripting for non-Bash programmers

If you think Bash is ugly, or inefficient, or underpowered, or any combination of those attributes, you might want to read this. Try to think of Bash as of super-terse programming language with a totally different way of doing things, and you might actually be surprised at how much fun it can be.

Beauty of Bash

The character is a Japanese ideogram for the word "beauty". It's probably cryptic to most of you, and you wouldn't be able to easily tell what it means if at all. However, it's just one character representing a whole word. If you are wondering why anyone would want to learn a few thousands of these in order to communicate with others, let me tell you something: it's efficient. You write a few of these, and you have a sentence. They also take care of ambiguity. Two words with exact same pronunciation have two completely different ideograms. They also combine well. Having two ideograms next to each other, they yield new meaning. In one word, they are terse.

Searching for beauty in programming languages, people end up in many different places. Some prefer verboseness, some prefer similarity with natural languages like English. Some prefer terseness.

Whether you find Bash beautiful or not, whether you think it's elegant or not, at some point you will have to agree that it is terse. It's terse in a very practical way, the way that ideograms are. To me, Bash's beauty comes from this very characteristic — from the fact that one character, which has no apparent meaning to people who don't know it, can represent a powerful concept that other languages cannot match with 3 lines of code. Oh, and have I mentioned "terse"?

Think "scripting the *NIX"

First of all, Bash can be used to program lots of different things, but it's the best tool for scripting entire systems. Think of it as of an API to script a *nix system, just as you would use an API to interact with a 3rd party service, or script a game.

Command | pipe > redirect

The first thing you should learn is pipes. This is the single most awesome feature of most *nix shells, and one that I consider the definitive difference between Bash and most other languages. It's the not-so-secret sauce that makes Bash scripts so short and maintainable.

Here's a piece of code written in Python:

f = open('file', 'r')
content = f.read()
f.close()
newcontent = ['foo ' + line for line in content.split('\n')]
f = open('anotherfile', 'w')
f.write('\n'.join(newcontent))
f.close()

Here is the same thing in Bash:

cat file | sed 's/^\(.*\)$/foo \1/' > anotherfile

Both programs do exactly the same thing, but Bash does that in a single line. It's the kind of terseness I keep mentioning. This is all possible thanks to pipes and redirects, and that fact that standard output is a file-like object, so you can reuse the same API to write to an actual file (by redirecting the output to it). Pipes, on the other hand, are like filters that you place between the final target (the output file) and the source of the stream (cat).

You also notice that in Bash, you access your files normally, like you would in your terminal. That's because Bash is your terminal shell as well, so anything you do in the shell, you can do in your scripts. There is no special "open file" thing in Bash.

Because you can have more than one pipe, you never do anything overly complex with them, so it makes you code simpler to write and read. Here is one example:

sub() {
    sed 's/'"$1"'/'"$2"'/'
}

remove() {
    grep -v "$1"
}

# Remove lines that start with "foo",
# replace "fam" with "bar", and
# replace curly brackets with square ones
cat file \
    | remove ^foo \
    | sub fam bar \
    | sub '{' '[' \
    | sub '}' ']' > newfile

Do you notice how you never have to refer to "current line" anywhere? Pipe is the implicit "work on current line". (Well, not always, but you'll learn the nuances as you do more Bash scripting.)

Single-quotes, double-quotes, un-quotes

Take another look at the sub function:

sed 's/'"$1"'/'"$2"'/'

What a weird array of quote characters, right? In Bash, it sometimes seems every character counts. At least with quotes, it's definitely true. Single quotes and double quotes mean completely different things in Bash. Single-quoted text is a literal string. Double-quoted text can have interpolated variables. Unquoted text is treated as multiple strings separated by spaces (actually, spaces is the default, but you can control the separator character).

One thing to note is that any quoted string is also part of an unquoted string. For example:

mv "file with spaces.txt" "another file with spaces.txt"

Whole of "file with spaces.txt" "another file with spaces.txt" is one unquoted string, which consists of two quoted strings separated by a single space.

In fact, quotes act like switches that tell Bash to interpret something differently from the point where it sees a quote character to the point where it sees a matching quote character. Let's go back to our sed example:

sed 's/'"$1"'/'"$2"'/'

Bash sees the single quote, and says "Ok, I'm in literal string now, so don't interpolate any variables." Then the single quote ends and double quotes start, so Bash decides to interpolate the variable but it interprets the contents of the double-quotes as a single string, and doesn't treat spaces as separators. This keeps sed from thinking it is now receiving another argument. And so on and so forth. This keeps the whole of the sed command as one whole string (because there are no spaces between adjacent quoted strings). It takes a short while to get your brain wrapped around this, but it's pretty awesome. It also implies that you don't have to use any special punctuation to separate arguments: space is the separator.

Let's take a look at another classic example:

col() {
    awk '{ print $'$1'; }'
}

At first glance, it may look as if though you have a quoted $1 embedded in the outer quotes, but $1 is actually sandwiched between two quoted strings. Awk uses the $something syntax to represent columns, so, by single-quoting it, we tell Bash to keep the $ character within the single quotes as a literal dollar character. On the other hand, we do want Bash to interpolate the actual $1 variable that is passed to the col() function, so we unquote that part.

This function is then usable in a pipe:

cat /var/log/mylog | col 2

When the argument passed to col() function (2 in our case) is interpolated, the resulting Awk command looks like this:

awk '{ print $2; }'

That prints the second column of the log file located at /var/log/mylog.

Everything is string

There aren't any special values (if you don't count container types like arrays). Everything is just strings. Numbers are strings too, used in a different way. For example, if you say myvar=2, 2 is just an unquoted string. It has no special meaning compared to foo or 'bar'.

Context is crucial to what some value is treated like. If you want to use some value as a number, you have to set the context. For eample:

varx=2
vary=3
echo $(( $varx + $vary ))

The $(()) syntax (yes, with double brackets) is used to perform arithmetic operations in Bash, and it's within these special brackets that variables are actually treated as numbers (no point in performing arithmetic on non-numbers).

Another case where you want numbers is in if conditionals:

if [ $varx -gt $vary ]; then
    echo "Is greater"
fi

The -gt (greater than) operator is strictly used for integer comparisons. Other operators are -eq (equal), -ne (not equal), -lt (less than), -le (less than or equal), and -ge (greater than or equal). The familiar comparison operators like ==, !=, >, or < are actually used to compare strings (according to their lexicographical sorting order).

Speaking of if conditionals, read more about it in Bash Beginners Guide. You'll be surprised at how many bells and whistles it has.

Capturing output of commands

You can capture the output of some command by using the $(...) notation. Here's an example:

myvar=$(cat somefile | col 2)

The $(), together with its contents, represents the output of the commands contained, basically just strings. You can process them as you would any variable. You can assign to variables, or put that in strings. For example:

rm "$(ls *.txt | grep -v '^[:alpha:]')"

The above example removes all files that have .txt extensions but whose filenames do not start with letters.

Trapping errors

By default, Bash scripts will plough through any errors in your script, be it programming errors, or errors thrown by commands you call. This may be fine in some situation (e.g., when failure is expected in some cases, and you just need the script to forget about it), but in most cases, you want the script to fail. Here is a simple trick to get that behaviour.

set -e

When Bash sees the above line (usually placed right at the top of the script), it will halt and break on all errors from that point on. That's all fine and dandy, but there is no try and catch (or try and except) so what do you do? To handle these errors, you can use the || (logical or) operator.

some_command || { echo "Ouch, error!"; exit 2; }
some_other_command || echo "This is ok, carry on!"

So why the logical or? Bash treats the 0 exit code returned by the program to mean "OK" (you could say it's equivalent of "true"), and any non-0 return code is essentially the equivalent of "false". So if the program returns a non-0, Bash will look for the expression on the other side of the logical or operator. The other side could be as simple or as complex as you like. You can write error handling functions, for example. Note that the other clause is only evaluated if the first one fails.

Regexp validation

The other day, I was trying to add some validation to one of my scripts, and caught myself trying to exercise Pythonism in Bash. Here's what I would normally do:

import re
import sys
input_re = re.compile(r'[a-z]{2,10}-[a-z]{3,10}')
if len(sys.argv) > 0 and not input_rere.match(sys.argv[1]):
   echo "Wrong answer!"
   sys.exit(1)
arg1 = sys.argv[1]

Here is how I do something like that in Bash:

myvar=$(echo $1 \
    | grep '[[:lower:]]\{2,10\}-[[:lower:]]\{3,10\}' \
    || echo '')
if [ -z "$myvar" ]; then
    echo "Wrong answer!"
    exit 1
fi

Essentially, I erase the value if it doesn't match the regexp and test if it is empty. If you think about it, it's not too different from what I normally do in Python, but I did have to think a little differently, and use regexp-capable commands instead of using a proper regexp data type. (Incidentally, I could have used sed and avoided the || echo '' part.)

You are not alone

You have already seen that in Bash scripts you are allowed to use any commands and programs that you have access to from the terminal. You've already seen sed and awk, which are not part of Bash. I'm sure you can guess the kind of power this gives you.

You should at least take a look at some of the other commonly used commands like grep, sort, tail, find, head, etc. They all come with manpages, and deliciously simple API, and the ability to participate in your pipe parties. And let's not forget tools like openSSH, curl, and similar, which give you access to Internet. Bash can also interact with your desktop environments using commands like notify-send (comes with notification daemon), and display pop-ups. In most languages, you would need wrapper libraries, or complex syntax for getting to these tools, but in Bash, these are at your disposal using the same syntax you normally use to access them in your terminal.

Want to learn | more

If you are interested in Bash after reading this article, but are not exactly sure about some of the details, you may want to start with the Bash Beginners Guide. Then you can move on the Advanced Bash Scripting, and give Sed - Introduction by Bruce Barnett a light read. And don't forget the manpages.