Bourne Shell Programming

Justification

Because, in a word, it's useful.

Many standard utilities (rdist, make, cron, etc.) allow you to specify a command to run at a certain time. Usually, this command is simply passed to the Bourne shell, which means that you can execute whole scripts, should you choose to do so.

Lastly, Unix runs Bourne shell scripts when it boots. If you want to modify the boot-time behavior of a system, you need to learn to write and modify Bourne shell scripts.

What's it all about?

Furthermore, and this is what this tutorial is all about, you can put commands in a file and execute them all at once. This is known as a script. Here's a simple one:

#!/bin/sh
# Rotate procmail log files
cd /homes/arensb/Mail
rm procmail.log.6       # This is redundant
mv procmail.log.5 procmail.log.6
mv procmail.log.4 procmail.log.5
mv procmail.log.3 procmail.log.4
mv procmail.log.2 procmail.log.3
mv procmail.log.1 procmail.log.2
mv procmail.log.0 procmail.log.1
mv procmail.log procmail.log.0

There are several things to note here: first of all, comments begin with a hash (#) and continue to the end of the line (the first line is special, and we'll cover that in just a moment).

Secondly, the script itself is just a series of commands. I use this script to rotate log files, as it says. I could just as easily have typed these commands in by hand, but I'm lazy, and I don't feel like it. Plus, if I did, I might make a typo at the wrong moment and really make a mess.

#!/bin/sh

A script, like any file that can be run as a command, needs to be executable: save this script as rotatelog and run

chmod +x rotatelog

./rotatelog

Unlike some other operating systems, Unix allows any program to be used as a script interpreter. This is why people talk about ``a Bourne shell script'' or ``an awk script.'' One might even write a more script, or an ls script (though the latter wouldn't be terribly useful). Hence, it is important to let Unix know which program will be interpreting the script.

When Unix tries to execute the script, it sees the first two characters (#!) and knows that it is a script. It then reads the rest of the line to find out which program is to execute the script. For a Bourne shell script, this will be /bin/sh. Hence, the first line of our script must be

#!/bin/sh

After the command interpreter, you can have one, and sometimes more, options. Some flavors of Unix only allow one, though, so don't assume that you can have more.

Once Unix has found out which program will be acting as the interpreter for the script, it runs that program, and passes it the name of the script on the command line. Thus, when you run ./rotatelog, it behaves exactly as if you had run /bin/sh ./rotatelog.

Variables

VAR=value

$VAR

${VAR}

The latter syntax is useful if the variable name is immediately followed by other text:

#!/bin/sh
COLOR=yellow

echo This looks $COLORish
echo This seems ${COLOR}ish

This looks
This seems yellowish

Local vs. environment variables

Environment variables are passed to subprocesses. Local variables are not.

By default, variables are local. To turn a local variable into an environment variable, use

export VAR

Here's a simple wrapper for a program:

#!/bin/sh
NETSCAPE_HOME=/usr/imports/libdata

CLASSPATH=$NETSCAPE_HOME/classes
export CLASSPATH

$NETSCAPE_HOME/bin/netscape.bin

The only way to unexport a variable is to unset it:

unset VAR

Also, note that if a variable was passed in as part of the environment, it is already an environment variable when your script starts running. If there is a variable that you really don't want to pass to any subprocesses, you should unset it near the top of your script. This is rare, but it might conceivably happen.

If you refer to a variable that hasn't been defined, sh substitutes the empty string.

#!/bin/sh
echo aaa $FOO bbb
echo xxx${FOO}yyy

aaa bbb
xxxyyy

Special variables

Command-line arguments

If you have more than nine command-line arguments, you can use the shift command: this discards the first command-line argument, and bumps the remaining ones up by one position: $2 becomes $1, $8 becomes $7, and so forth.

The variable $0 (zero) contains the name of the script (argv[0] in C programs).

Often, it is useful to just list all of the command-line arguments. For this, sh provides the variables $* (star) and $@ (at). Each of these expands to a string containing all of the command-line arguments, as if you had used $1 $2 $3...

The difference between $* and $@ lies in the way they behave when they occur inside double quotes: $* behaves in the normal way, whereas $@ creates a separate double-quoted string for each command-line argument. That is, "$*" behaves as if you had written "$1 $2 $3", whereas "$@" behaves as if you had written "$1" "$2" "$3".

Finally, $# contains the number of command-line arguments that were given.

Other special variables

Quasi-variable constructs

${VAR:-expression}

Use default value: if VAR is set and non-null, expands to $VAR. Otherwise, expands to expression.

${VAR:=expression}

Set default value: if VAR is set and non-null, expands to $VAR. Otherwise, sets VAR to expression and expands to expression.

${VAR:?[expression]}

If VAR is set and non-null, expands to $VAR. Otherwise, prints expression to standard error and exits with a non-zero exit status.

${VAR:+expression}

If VAR is set and non-null, expands to the empty string. Otherwise, expands to expression.

${#VAR}

Expands to the length of $VAR.

The above patterns test whether VAR is set and non-null. Without the colon, they only test whether VAR is set.

Patterns and Globbing

*

Matches zero or more characters.

?

Matches exactly one character.

[range]

Matches any character in range. range can be either a list of characters that match, or two endpoints separated by a dash: [ak3] matches either a, k, or 3; [a-z] matches any character in the range a through z; [a-mz] matches either a character in the range a through m, or z. If you wish to include a dash as part of the range, it must be the first character, e.g., [-p] will match either a dash or p.

As a special case, when a glob begins with * or ?, it does not match files that begin with a dot. To match these, you need to specify the dot explicitly (e.g., .*, /tmp/.*).

Note to MS-DOS users: under MS-DOS, the pattern *.* matches every file. In sh, it matches every file that contains a dot.

Quoting

echo * MAKE   $$$   FAST *

sh supports several types of quotes. Which one you use depends on what you want to do.

Backslash

The backslash is itself special, so to escape it, just double it: \\.

Single quotes

'foo'

echo '* MAKE   $$$   FAST *'

Note that a backslash inside single quotes also loses its special meaning, so you don't need to double it. There is no way to have a single quote inside single quotes.

Double quotes

"foo"

Backquotes

`cmd`

echo You are `whoami`

You are arensb

Built-in commands

{ commands ; }, ( commands )

Execute commands in a subshell. That is, run them as if they were a single command. This is useful when I/O redirection is involved, since you can pipe data to or from a mini-script inside a pipeline.

The { commands; } variant is somewhat more efficient, since it doesn't spawn a true subshell. This also means that if you set variables inside of it, the changes will be visible in the rest of the script.

: (colon)

Does nothing. This is generally seen as

: ${VAR:=default}

. filename

The dot command reads in the specified filename, as if it had occurred at that place in the script.

bg [job], fg [job]

bg runs the specified job (or the current job, if none is specified) in the background. fg resumes the specified job (or the current job, if none is specified) in the foreground. Jobs are specified as %number. The jobs command lists jobs.

cd [dir]

Sets the current directory to dir. If dir is not specified, sets the current directory to the home directory.

pwd

Prints the current directory.

echo args

Prints args to standard output.

eval args

Evaluates args as a sh expression. This allows you to construct a string on the fly (e.g., using a variable that holds the name of a variable that you want to set) and execute it.

exec command

Runs the specified command, and replaces the current shell with it. That is, nothing after the exec statement will be executed, unless the exec itself fails.

exit [n]

Exit the current shell with exit code n. This defaults to zero.

kill [-sig] %job

Send signal sig to the specified job. sig can be either numeric or symbolic. kill -l lists all available signals. By default, sig is SIGTERM (15).

read name...

Reads one line from standard input and assigns it to the variable name. If several variables name1, name2, name3 etc. are specified, then the first word of the line read is assigned to name1, the second to name2, and so forth. Any remaining words are assigned to the last variable.

set [+/-flag] [arg]

With no arguments, prints the values of all variables.

set -x turns on the x option to sh; set +x turns it off.

set args... sets the command-line arguments to args.

test expression

Evaluates a boolean expression and exits with an exit code of zero if it is true, or non-zero if it is false. See test for more details.

trap [command sig]...

If signal sig is sent to the shell, execute command. This is useful for exiting cleanly (e.g., removing temporary files etc.) when the script is interrupted.

ulimit

Print or set system limits on resource usage.

umask [nnn]

Sets the umask to nnn (an octal number). With no argument, prints the current umask. This is most useful when you want to create files, but want to restrict who can read or write them.

wait [n]

Wait for the background process whose PID is n to terminate. With no arguments, waits for all of the background processes to terminate.

Flow control

if

if condition ; then
        commands
[elif condition ; then
        commands]...
[else
        commands]
fi

The if statement pretty much does what you'd expect: if condition is true, it executes the if-block. Otherwise, it executes the else-block, if there is one. The elif construct is just syntactic sugar, to let you avoid nesting multiple if statements.

#!/bin/sh
myname=`whoami`

if [ $myname = root ]; then
        echo "Welcome to FooSoft 3.0"
else
        echo "You must be root to run this script"
        exit 1
fi

Actually, I didn't: [ is actually a command, /bin/[, and is another name for the test command.

This is why you shouldn't call a test program test: if you have ``.'' at the end of your path, as you should, executing test will run /bin/test.

See below for details.

The condition can actually be any command. If it returns a zero exit status, the condition is true; otherwise, it is false. Thus, you can write things like

#!/bin/sh
user=arnie
if grep $user /etc/passwd; then
        echo "$user has an account"
else
        echo "$user doesn't have an account"
fi

while

while condition; do
        commands
done

As you might expect, the while loop executes commands as long as condition is true. Again, condition can be any command, and is true if the command exits with a zero exit status.

A while loop may contain two special commands: break and continue.

break exits the while loop immediately, jumping to the next statement after the done.

continue skips the rest of the body of the loop, and jumps back to the top, to where condition is evaluated.

for

for var in list; do
        commands
done

list is zero or more words. The for construct will assign the variable var to each word in turn, then execute commands. For example:

#!/bin/sh
for i in foo bar baz "do be do"; do
        echo "$i"
done

foo
bar
baz
do be do

case

case expression in
        pattern)
                commands
                ;;
        ...
esac

pattern is a glob pattern (see globbing).

The patterns are evaluated in the order in which they are seen, and only the first pattern that matches will be executed. Often, you'll want to include a ``none of the above'' clause; to do this, use * as your last pattern.

I/O redirection

However, one can do many interesting things by redirecting one or more file descriptor:

< filename

Connect standard input to the file filename. This allows you to have a command read from the file, rather than having to type its input in by hand.

> filename

Connect standard output to the file filename. This allows you to save the output of a command to a file. If the file does not exist, it is created. If it does exist, it is emptied before anything happens.

(Exercise: why doesn't cat * > zzzzzzz work the way you'd expect?)

>> filename

Connects standard output to the file filename. Unlike >, however, the output of the command is appended to filename.

<<word

This construct isn't used nearly as often as it could be. It causes the command's standard input to come from... standard input, but only until word appears on a line by itself. Note that there is no space between << and word.

This can be used as a mini-file within a script, e.g.,

	cat > foo.c <<EOT
	#include <stdio.h>

	main()
	{
		printf("Hello, world!\n");
	}
	EOT
	

It is also useful for printing multiline messages, e.g.:

	line=13
	cat <<EOT
	An error occurred on line $line.
	See page 98 of the manual for details.
	EOT
	

As this example shows, by default, << acts like double quotes (i.e., variables are expanded). If, however, word is quoted, then << acts like single quotes.

<&digit

Use file descriptor digit as standard input.

>&digit

Use file descriptor digit as standard output.

<&-

Close standard input.

>&-

Close standard output.

command1 | command2

Creates a pipeline: the standard output of command1 is connected to the standard input of command2. This is functionally identical to

command1 > /tmp/foo
command2 < /tmp/foo

except that no temporary file is created, and both commands can run at the same time

There is a proverb that says, ``A temporary file is just a pipe with an attitude and a will to live.''

Any number of commands can be pipelined together.

command1 && command2

Execute command1. Then, if it exited with a zero (true) exit status, execute command2.

command1 || command2

Execute command1. Then, if it exited with a non-zero (false) exit status, execute command2.

command 2>&1 > filename

This is also useful for printing error messages:

echo "Danger! Danger Will Robinson!" 1>&2

Note that I/O redirections are parsed in the order they are encountered, from left to right. This allows you to do fairly tricky things, including throwing out standard output, and piping standard output to a command.

Functions

A function is defined using

name () {
        commands
}

name args...

You can redirect a function's I/O, embed it in backquotes, etc., just like any other command.

One way in which functions differ from external scripts is that the shell does not spawn a subshell to execute them. This means that if you set a variable inside a function, the new value will be visible outside of the function.

A function can use return n to terminate with an exit status of n. Obviously, it can also exit n, but that would terminate the entire script.

Function arguments

Useful utilities

basename

basename /foo/bar/baz

baz

dirname

dirname /foo/bar/baz

/foo/bar

[

If test is invoked as [, then it requires a closing bracket ] as its last argument. Otherwise, there must be no closing bracket.

test understands the following expressions, among others:

-e filename

True if filename exists.

-d filename

True if filename exists and is a directory.

-f filename

True if filename exists and is a plain file.

-h filename

True if filename exists and is a symbolic link.

-r filename

True if filename exists and is readable.

-w filename

True if filename exists and is writable.

-n string

True if the length of string is non-zero.

-z string

True if the length of string is zero.

string

True if string is not the empty string.

s1 = s2

True if the strings s1 and s2 are identical.

s1 != s2

True if the strings s1 and s2 are not identical.

n1 -eq n2

True if the numbers n1 and n2 are equal.

n1 -ne n2

True if the numbers n1 and n2 are not equal.

n1 -gt n2

True if the number n1 is greater than n2.

n1 -ge n2

True if the number n1 is greater than or equal to n2.

n1 -lt n2

True if the number n1 is less than n2.

n1 -le n2

True if the number n1 is less than or equal to n2.

! expression

Negates expression, that is, returns true iff expression is false.

expr1 -a expr2

True if both expressions, expr1 and expr2 are true.

expr1 -o expr2

True if either expression, expr1 or expr2 is true.

( expression )

True if expression is true. This allows one to nest expressions.

echo

echo simply prints its arguments to standard output. It can also be told not to append a newline at the end: under BSD-like flavors of Unix, use

echo -n "string"

echo "string\c"

awk

awk -F : '{print $1, $3 }' /etc/passwd

The -F : option says that the input records are separated by colons. By default, awk uses whitespace as the field separator.

sed

sed -e 's/foo/bar/g'

The trailing g says to replace all instances of ``foo'' with ``bar'' on a line. Without it, only the first instance would be replaced.

tee

By default, tee empties filename before it begins. With the -a option, it appends to filename.

Debugging

The -n option causes sh to read the script but not execute any commands. This is useful for checking syntax.

The -x option causes sh to print each command to standard error before executing it. Since this can generate a lot of output, you may want to turn tracing on just before the section that you want to trace, and turn it off immediately afterward:

set -x
# XXX - What's wrong with this code?
grep $user /etc/passwd 1>&2 > /dev/null
set +x

Style

Prefer simple, linear application

Therefore, it is best to keep things simple and linear: do A, then do B, then do C, and exit. If you find yourself writing many nested loops, or building awk scripts on the fly, you're probably better off rewriting it in Perl or C.

Put customization variables at the top

#!/bin/sh
...300 lines further down...
/usr/local/bin/gmake foo

#!/bin/sh
GMAKE=/usr/local/bin/gmake
...300 lines further down...
$GMAKE foo

Don't go overboard with functions

Multi-line strings

Use : ${VAR:=value} to set defaults

vi $filename

However,

$VISUAL $filename

So use

: ${VISUAL:=vi}
$VISUAL $filename

Paranoia

Setuid scripts

As we saw above, the way scripts work, Unix opens the file to find out which program will be the file's interpreter. It then invokes the interpeter, and passes it the script's pathname as a command line argument. The interpreter then opens the file, reads it, and executes it.

From the above, you can see that there is a delay between when the OS opens the script, and when the interpreter opens it. This means that there is a race condition that an attacker can exploit: create a symlink that points to the setuid script; then, after the OS has determined the interpeter, but before the interpreter opens the file, replace that symlink with some other script of your choice. Presto! Instant root shell!

This problem is inherent to the way scripts are processed, and therefore cannot easily be fixed.

Compiled programs do not suffer from this problem, since a.out (compiled executable) files are not closed then reopened, but directly loaded into memory. Hence, if you have an application that needs to be setuid, but is most easily written as a script, you can write a wrapper in C that simply exec()s the script. You still need to watch out for the usual problems that involve writing setuid programs, and you have to be paranoid when writing your script, but all of these problems are surmountable. The double-open problem is not.

$IFS

IFS=

$PATH

In particular, the user might have ``.'' (dot) as the first element of his path, and put a program called ls or grep in the current directory, with disastrous results.

In general, never put ``.'' or any other relative directory on your path.

I like to begin by putting the line

PATH=

Quoted variables

I once had a script fail because a user had put a square bracket in his GCOS field in /etc/passwd. You're best off just quoting everything, unless you know for sure that you shouldn't.

Potentially unset variables

if [ $answer = yes ]; then

However, $answer might be set to the empty string, so sh would see if [ = yes ]; then, which would cause an error. Better to write

if [ "$answer" = yes ]; then

The danger here is that $answer might be set to -f, so sh would see if [ -f = yes ]; then, which would also cause an error.

Therefore, write

if [ x"$answer" = xyes ]; then

What about the C shell?