The Unix Shell

Files and Directories

The part of the operating system responsible for managing files and directories is called the file system. It organizes our data into files, which hold information, and directories (also called “folders”, for example, on Windows systems), which hold files or other directories.

Several commands are frequently used to create, inspect, rename, and delete files and directories. To start exploring them, let’s open a shell window:

$

The dollar sign is a prompt, which represents our input interface to the shell. It shows us that the shell is waiting for input; your shell may show something more elaborate.

Type the command whoami, then press the Enter key (sometimes called Return) to send the command to the shell. The command’s output is the ID of the current user, i.e., it shows us who the shell thinks we are:

$ whoami

nelle

More specifically, when we type whoami the shell:

1. finds a program called whoami,
2. runs that program,
3. displays that program’s output, then
4. displays a new prompt to tell us that it’s ready for more commands.

Next, let’s find out where we are by running a command called pwd (which stands for “print working directory”). At any moment, our current working directory is our current default directory, i.e., the directory that the computer assumes we want to run commands in unless we explicitly specify something else. Here, the computer’s response is /Users/nelle, which is Nelle’s home directory:

$ pwd

/Users/nelle

To understand what a “home directory” is, let’s have a look at how the file system as a whole is organized. At the top is the root directory that holds everything else. We refer to it using a slash character / on its own; this is the leading slash in /Users/nelle.

Inside that directory are several other directories, e.g.: bin (which is where some built-in programs are stored), data (for miscellaneous data files), Users (where users’ personal directories are located), tmp (for temporary files that don’t need to be stored long-term), and so on:

The File System

We know that our current working directory /Users/nelle is stored inside /Users because /Users is the first part of its name. Similarly, we know that /Users is stored inside the root directory / because its name begins with /.

Underneath /Users, we find one directory for each user with an account on this machine, e.g.: /Users/imhotep, /Users/larry, and ours in /Users/nelle, which is why nelle is the last part of the directory’s name.

Home Directories

Let’s see what’s in our home directory by running ls, which stands for “listing” (the ... refers to other files and directories that have been left out for clarity):

$ ls

2015-12-15-southampton Misc                   Solar.pdf
Applications           Movies                 Teaching
Desktop                Music                  ThunderbirdTemp
Development            Notes.txt              VirtualBox VMs
Documents              Pictures               bin
Downloads              Pizza.cfg              mbox
...

Of course, this listing will depend on what you have in your own home directory.

We need to get into the repository directory 2015-12-15-southampton, so what if we want to change our current working directory? Before we do this, pwd shows us that we’re in /Users/nelle.

$ pwd

/Users/nelle

We can use cd followed by a directory name to change our working directory. cd stands for “change directory”, which is a bit misleading: the command doesn’t change the directory, it changes the shell’s idea of what directory we are in.

$ cd 2015-12-15-southampton

cd doesn’t print anything, but if we run pwd after it, we can see that we are now in /Users/nelle/2015-12-15-southampton. If we run ls without arguments now, it lists the contents of /Users/nelle/2015-12-15-southampton, because that’s where we now are:

$ pwd

/Users/nelle/2015-12-15-southampton

ls prints the names of the files and directories in the current directory in alphabetical order, arranged neatly into columns (where there is space to do so). We can make its output more comprehensible by using the flag -F, which tells ls to add a trailing / to the names of directories:

$ ls -F

CONDUCT.md              _config.yml             prerequisites.html
CONTRIBUTING.md         _includes/              prerequisites.md
CUSTOMIZATION.md        _layouts/               reference.html
DESIGN.md               css/                    schedule.html
FAQ.md                  deck.js/                setup/
LICENSE.md              index.html              template/
Makefile                js/                     tools/
README.md               novice/                 welcome/
SETUP.md                prerequisites-ref.html

Here, we can see that this directory contains a number of sub-directories. The names that don’t have trailing slashes, like reference.html, prerequisites.md, and preqrequisites.pdf, are plain old files. And note that there is a space between ls and -F: without it, the shell thinks we’re trying to run a command called ls-F, which doesn’t exist.

For this exercise, we need to change our working directory to novice, and then shell (within the novice directory). We can do this by:

$ cd novice/shell

Note that we are able to add directories together by using /. Now if we view the contents of that directory:

$ ls -F

00-intro.html          AUTHORS                img/
00-intro.md            CONDUCT.md             index.html
01-filedir.html        CONTRIBUTING.md        index.md
01-filedir.md          LICENSE.html           instructors.html
02-create.html         LICENSE.md             instructors.md
02-create.md           Makefile               js/
03-pipefilter.html     README.md              motivation.html
03-pipefilter.md       _includes/             motivation.md
04-loop.html           _layouts/              reference.html
04-loop.md             css/                   reference.md
05-script.html         data/                  requirements.txt
05-script.md           discussion.html        shell-novice-data.zip
06-find.html           discussion.md          test_directory/
06-find.md             fig/                   tools/

Note that under Git Bash in Windows, the / is appended automatically.

Now let’s take a look at what’s in the directory test_directory, by running ls -F test_directory.

i.e., the command ls with the arguments -F and test_directory. The second argument — the one without a leading dash — tells ls that we want a listing of something other than our current working directory:

$ ls -F test_directory

creatures/          molecules/          notes.txt           solar.pdf
data/               north-pacific-gyre/ pizza.cfg           writing/

The output shows us that there are some files and sub-directories. Organising things hierarchically in this way helps us keep track of our work: it’s a bit like using a filing cabinet to store things. It’s possible to put hundreds of files in our home directory, for example, just as it’s possible to pile hundreds of printed papers on our desk, but it’s a self-defeating strategy.

Notice, by the way, that we spelled the directory name test_directory. It doesn’t have a trailing slash: that’s added to directory names by ls when we use the -F flag to help us tell things apart. And it doesn’t begin with a slash because it’s a relative path, i.e., it tells ls how to find something from where we are, rather than from the root of the file system.

If we run ls -F /test_directory (with a leading slash) we get a different response, because /test_directory is an absolute path:

$ ls -F /test_directory

ls: /test_directory: No such file or directory

The leading / tells the computer to follow the path from the root of the file system, so it always refers to exactly one directory, no matter where we are when we run the command. In this case, there is no data directory in the root of the file system.

Typing ls -F test_directory is a bit painful, so a handy shortcut is to type in the first few letters and press the TAB key, e.g.

$ ls -F tes

Pressing TAB, the shell automatically completes the directory name:

$ ls -F test_directory/

This is known as tab completion on any matches with those first few letters. If there are more than one files or directories that match those letters, the shell will show you both — you can then enter more characters (then using TAB again) until it is able to identify the precise file you want and finish the tab completion.

Let’s change our directory to test_directory:

$ cd test_directory

We know how to go down the directory tree: but how do we go up? We could use an absolute path, e.g. cd /Users/nelle/2015-12-15-southampton/novice/shell.

but it’s almost always simpler to use cd .. to go up one level:

$ pwd

/Users/nelle/2015-12-15-southampton/novice/shell/test_directory

$ cd ..

.. is a special directory name meaning “the directory containing this one”, or more succinctly, the parent of the current directory.

$ pwd

/Users/nelle/2015-12-15-southampton/novice/shell/

Let’s go back into our test directory:

$ cd test_directory

The special directory .. doesn’t usually show up when we run ls. If we want to display it, we can give ls the -a flag:

$ ls -F -a

./                  data/               notes.txt           writing/
../                 molecules/          pizza.cfg
creatures/          north-pacific-gyre/ solar.pdf

-a stands for “show all”; it forces ls to show us file and directory names that begin with ., such as .. (which, if we’re in /Users/nelle/2015-12-15-southampton/shell/novice/test_directory, refers to the /Users/nelle/2015-12-15-southampton/2015-12-15-southampton/novice/shell/ directory). As you can see, it also displays another special directory that’s just called ., which means “the current working directory”. It may seem redundant to have a name for it, but we’ll see some uses for it soon.

Another handy feature is that we can reference our home directory with ~, e.g.:

$ ls ~/2015-12-15-southampton

CONDUCT.md              _config.yml             prerequisites.html
CONTRIBUTING.md         _includes/              prerequisites.md
CUSTOMIZATION.md        _layouts/               reference.html
DESIGN.md               css/                    schedule.html
FAQ.md                  deck.js/                setup/
LICENSE.md              index.html              template/
Makefile                js/                     tools/
README.md               novice/                 welcome/
SETUP.md                prerequisites-ref.html

Which again shows us our repository directory.

File System for Challenge Questions

pnas-sub/ pnas-final/ original/