The Bash Shell

Loops

Wildcards and tab completion are two ways to reduce typing as well as typing mistakes. Another is to tell the shell to do something over and over again, which could save us considerable time, depending on how many times we need the shell to do this thing.

Couldn’t we just…

Suppose we have several hundred genome data files named basilisk.dat, minotaur.dat, unicorn.dat, and so on. In this example, we’ll use the test_directory/creatures directory which only has three example files, but the principles can be applied to many many more files at once. Let’s first go to the creatures directory (using tab completion to enter the full directory will save considerable typing here!):

$ cd ~/2020-10-29-socobio-crs/novice/shell/test_directory/creatures
$ ls

basilisk.dat minotaur.dat unicorn.dat

We would like to modify these files, but also save a version of the original files and rename them as original-basilisk.dat, original-minotaur.dat, original-unicorn.dat. We can’t use the following (don’t type this, it’s just for illustrative purposes):

$ mv *.dat original-*.dat

Because as we learnt previously, with wildcards that would expand to:

$ mv basilisk.dat minotaur.dat unicorn.dat original-*.dat

This wouldn’t back up our files, instead we would get an error. If on a Mac or Linux it would look like:

mv: target `original-*.dat' is not a directory

Or if on Windows using Git Bash, we would see:

usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory

Even though the error is different, the cause is the same. It arises when mv receives more than two inputs. When this happens, it expects the last input to be a directory where it can move all the files it was passed. Since there is no directory named original-*.dat in the creatures directory we get an error.

Using a loop to do something multiple times

Instead, we can use a loop to do some operation once for each thing in a list. Here’s a simple example that displays the first three lines of each file in turn.

Let’s create a new shell script using nano called top.sh that uses a loop.

$ nano top.sh

In that file enter the following:

for filename in basilisk.dat minotaur.dat unicorn.dat
do
    head -3 $filename
done

After saving it by using Control-O and Control-X, run the script:

$ bash top.sh

COMMON NAME: basilisk
CLASSIFICATION: basiliscus vulgaris
UPDATED: 1745-05-02
COMMON NAME: minotaur
CLASSIFICATION: minotaurus maximus
UPDATED: 1764-09-12
COMMON NAME: unicorn
CLASSIFICATION: equus monoceros
UPDATED: 1738-11-24

So what’s happening, and how does the loop work?

When the shell sees the keyword for, it knows it is supposed to repeat a command (or group of commands) once for each thing in a list. In this case, the list is the three filenames. Each time through the loop, the name of the thing currently being operated on is assigned to the variable called filename.

Inside the loop, we get the variable’s value by putting $ in front of it: $filename is basilisk.dat the first time through the loop, minotaur.dat the second, unicorn.dat the third, and so on.

By using the dollar sign we are telling the shell interpreter to treat filename as a variable name and substitute its value on its place, but not as some text or external command. When using variables it is also possible to put the names into curly braces to clearly delimit the variable name: $filename is equivalent to ${filename}, but is different from ${file}name. You may find this notation in other people’s programs.

Finally, the command that’s actually being run is our old friend head, so this loop prints out the first three lines of each data file in turn.

Why the extra spaces?

Note the use of spaces for indentation before the head command. This line is part of the body of the loop (the part that gets executed many times) and whilst extra spaces don’t affect how the script runs, it is considered best practice to use indentation to highlight the loop body. In general programming, indentation is very important. Without indentation in code blocks such as these,, code becomes much harder to read.

Dos and don’ts of variable naming

We have called the variable in this loop filename in order to make its purpose clearer to human readers. The shell itself doesn’t care what the variable is called; if we wrote this loop as:

for x in basilisk.dat minotaur.dat unicorn.dat
do
    head -3 $x
done

or:

for temperature in basilisk.dat minotaur.dat unicorn.dat
do
    head -3 $temperature
done

it would work exactly the same way. Don’t do this. Programs are only useful if people can understand them, so meaningless names like x, or misleading names like temperature, increase the odds that the program won’t do what its readers think it does.

Looping over arbitrary numbers of files

Let’s assume there are many more of these .dat files. How would we run a loop over them all? Here’s a slightly more complicated loop to try next. Change our top.sh script to the following:

for filename in *.dat
do
    echo $filename
    head -100 $filename | tail -20
done

Save this file and exit nano.

The shell starts by expanding *.dat to create the list of files it will process, since with the * wildcard, this pattern will match anything that ends with .dat. The loop body then executes two commands for each of those files. The first, echo, just prints its command-line parameters to standard output. For example:

$ echo hello there

prints:

hello there

In this case, since the shell expands $filename to be the name of a file, echo $filename just prints the name of the file. Note that we can’t write this as:

for filename in *.dat
do
    $filename
    head -100 $filename | tail -20
done

because then the first time through the loop, when $filename expanded to basilisk.dat, the shell would try to run basilisk.dat as a program. Finally, the head and tail combination selects lines 81-100 from whatever file is being processed. Run this revised script now:

$ bash top.sh

And you should see (the ... indicates more gene sequences that appear in the output, but are omitted for clarity):

basilisk.dat
CGGTACCGAA
AAGGGTCGCG
CAAGTGTTCC
CGGGACAATA
GTTCTGCTAA
...
minotaur.dat
TAGGTTATAA
GGCACAACCG
CTTCACTGTA
GAGGTGTACA
AGGATCCGTT
...
unicorn.dat
CGGTACCGAA
AAGGGTCGCG
CAAGTGTTCC
CGGGACAATA
GTTCTGCTAA
...

File renaming revisited

Going back to our original file renaming problem, using what we’ve learnt we can solve it using the following loop. In a new script called rename.sh enter the following:

for filename in *.dat
do
    mv $filename original-$filename
done

This loop runs the mv command once for each filename. The first time, when $filename expands to basilisk.dat, the shell executes:

mv basilisk.dat original-basilisk.dat

The second time, the command is:

mv minotaur.dat original-minotaur.dat

The third time, the command is:

mv unicorn.dat original-unicorn.dat

Note that once you’ve run this command once, running it again has an interesting effect that we likely don’t intend - the .dat files we end up with are:

original-original-basilisk.dat original-original-unicorn.dat
original-original-minotaur.dat

This is because the .dat files picked up by for filename in *.dat will now match on original-basilisk.dat, original-unicorn.dat, and original-minotaur.dat, and each of these files is then renamed with yet another original- prefix added to it. This is another example of why you should always ensure you have a backup of files before you operate on them!

Exercises

fructose.dat    glucose.dat   sucrose.dat

for datafile in *.dat
do
    ls *.dat
done

for datafile in *.dat
do
  ls $datafile
done

for sugar in *.dat
do
    echo $sugar
    cat $sugar > xylose.dat
done

fructose.dat    glucose.dat   sucrose.dat   maltose.txt

for datafile in *.dat
do
    cat $datafile >> sugar.dat
done

for file in *.dat
do
  analyze $file > analyzed-$file
done

# Version 1
for file in *.dat
do
  echo analyze $file > analyzed-$file
done

# Version 2
for file in *.dat
do
  echo "analyze $file > analyzed-$file"
done

Next: Finding Things