Introduction

• Perl/Python may not be available on the machine.
• Boot init code may not be able to access Perl or Python
• Many existing scripts are shellscripts

This document covers only the sh/bash shells, although others are somewhat similar. I had only Linux bash (Mandrake 7.0) available for testing, but my memory of HP UX tells me that it's pretty applicable across Unices.

Note: Some nice sed one liners can be had at http://www-h.eng.cam.ac.uk/help/tpl/unix/sed.html.

Acknowledgments

Special thanks go out to Tony Becker, whose incredible bash presentation illuminated the fact that the left bracket character is really a symbolic link to /usr/bin/test command. After seeing Tony's presentation, I decided to start using more shellscripts and to write this web page. Mark Alexander and Nickolai Zeldovich helped me with shellscript file I/O.

And of course, thanks to the help and support of Troubleshooters.Com's visitors.

Hello World

echo "Hello World"

chmod a+x hello.sh

./hello.sh

Some Syntax Gotchas

if test "$currdir" = "/home/john"; then

[slitt@mydesk bin]$ ls -l [
lrwxrwxrwx    1 root     root            4 Apr 25 17:01 [ -> test*
[slitt@mydesk bin]$

Outputting to Stdout

aname="Florida"
let adollars=3290
bname="Utah"
let bdollars=817
printf "%10s budget is %06d\n" $aname $adollars
printf "%10s budget is %06d\n" $bname $bdollars

The preceding code prints the following:
 

[slitt@mydesk slitt]$ ./myscript    Florida budget is 003290       Utah budget is 000817 [slitt@mydesk slitt]$

As you can see, fields are properly padded. This is very similar to the C printf command.

Outputting to Stderr

echo "Hello World"
echo 1>&2 "This is an error"

Running this script you'll see both lines. But if you do the following:

./hello.sh | less

You can observe stderr in isolation with the following command, which diverts stdout from the screen:

./hello.sh > /dev/null

Arguments, Environment Variables and Return Values

There is other info that can be accessed from inside the script. For complete info, search the string Special Parameters in the bash man page. For another interesting topic, search Brace Expansion .

Saving PID of Spawned Binary Executable

Any running program appears in a ps ax command, and you can certainly grep for the executable. The problem is you don't know which process is the one spawned by a particular invocation of your shellscript. But if you can use the shellscript's process id (PID) as an argument to the binary executable, you can uniquely "brand" the executable, thus grep for the one and only one copy run from the particular shellscript.

Check out this code:
 
 

open less jj $$     #run job with script pid as arg ## Find job non-grep job with command and script pid ## jobline=`ps ax | grep -v grep | grep "less jj $$"` job=${jobline%% *}  #spawned pid is everything before 1st space echo $job

The preceding prints the binary executable's PID to stdout. It could also be redirected to a file for safe keeping. The first stsep is to run the command with the shellscript's PID ($$) as an argument. This is safe if the command ignores extraneous arguments. The next step is to find the job in a ps ax listing. Assuming $$ expands to 43210, you can grep for "less jj 43210 to find the executable. Except the grep will also return the grep command itself. So you filter out any commands containing the word "grep" with grep -v grep. This yields the single line containing the spawned binary executable's process. The number at the start of the line is the spawned executable's PID, and everything right of that point is extraneous. So you blow it off with the ${jobline%% *} construct (explained in the section on string manipulation). What's left is the PID of the spawned binary executable. I like it.

Environment Variables

echo $PWD
echo $PATH

You use the dollar sign to *read* an environment variable, but not to write one. The dollar sign can be though of as a "what is the value of" operation.

To write an environment variable, simply use the name of the variable on the left side of an equal sign (no spaces please). For instance, the following script creates an environment variable called SHELLSCRIPTS and sets it to the string "are cool"
 

echo \>$@\< echo $SHELLSCRIPTS SHELLSCRIPTS="are cool" echo $SHELLSCRIPTS

Once again, use the dollar sign only when you want to *use* the value of the variable -- you do not use the dollar sign when you want to *set* the value of the variable.

You'll notice that the new environment variable is in scope only within the shellscript. This can be proven by running the following command at the shell prompt:

set | grep SHELLSCRIPTS

. ./hello.sh

source ./hello.sh

SHELLSCRIPTS="set by parent"
echo "In parent, value is >$SHELLSCRIPTS<"
./child.sh

echo "In child, value is >$SHELLSCRIPTS<"

Look what happens when you run it:
 

[slitt@mydesk slitt]$ ./parent.sh In parent, value is >set by parent< In child, value is >< [slitt@mydesk slitt]$

The setting didn't survive the call to child.sh. If you want to pass the parent value to the child, use the export command in parent.sh, as shown below:
 

parent.sh

SHELLSCRIPTS="set by parent" echo "In parent, value is >$SHELLSCRIPTS<" export SHELLSCRIPTS ./child.sh

Now the value is passed to the child:
 

[slitt@mydesk slitt]$ ./parent.sh In parent, value is >set by parent< In child, value is >set by parent< [slitt@mydesk slitt]$

Can the child change the variable and send it back to the parent? Let's modify each to find out:
 

parent.sh

SHELLSCRIPTS="set by parent" echo "In parent, value is >$SHELLSCRIPTS<" export SHELLSCRIPTS ./child.sh echo "In parent after child, value is >$SHELLSCRIPTS<"

child.sh

echo "In child, value is >$SHELLSCRIPTS<" SHELLSCRIPTS="changed by child" echo "In child after change, value is >$SHELLSCRIPTS<"

[slitt@mydesk slitt]$ ./parent.sh In parent, value is >set by parent< In child, value is >set by parent< In child after change, value is >changed by child< In parent after child, value is >set by parent< [slitt@mydesk slitt]$

Oops! The child can't pass back its changes. In fact, there is *nothing* the *child* can do to pass back the change. Only the parent has the power to request the passback. That is done via the dot command or the source command. To use it, in parent.sh the call to child.sh would be changed to one of these two:

. ./child.sh

source ./child.sh

[slitt@mydesk slitt]$ ./parent.sh
In parent, value is >set by parent<
In child, value is >set by parent<
In child after change, value is >changed by child<
In parent after child, value is >changed by child<
[slitt@mydesk slitt]$

Returning a value

The following script, called showreturn.sh, can be used to show the return status of a command:
 

$@ echo Return value is $?

Running it on hello.sh confirms that the default return is 0:

[slitt@mydesk slitt]$ ./showreturn.sh ./hello.sh Hello World Return value is 0 [slitt@mydesk slitt]$

To return a specific return value, use the exit command as added to hello.sh below:
 

echo "Hello World" exit 44

[slitt@mydesk slitt]$ ./showreturn.sh ./hello.sh Hello World Return value is 44 [slitt@mydesk slitt]$

 

IFCONFIG_RETURN=$?

Summary

You can deduce the PID of a spawned executable by using the present shellscript's PID as an argument to the executable, and grepping for it in a ps ax command.

There is other info that can be accessed from inside the script. For complete info, search the string Special Parameters in the bash man page. For another interesting topic, search Brace Expansion.

The value of an environment variable is read by preceding the variable name with a dollar sign. However, the dollar sign is not used when setting the value of the variable. Instead, the variable name is immediately followed by an equal sign and the desired value. Setting an environment variable within a script only effects the value within that script, not in the calling parent or any called children. To have it affect child processes, you must use the export command before issuing the call to the child. A script *cannot* affect environment variable values of its calling parent without the cooperation of the parent. That parent cooperation is in the form of calling the script with a preceding dot and space, or with the source keyword.

By default, scripts return a value of 0. To return a different value, use the exit command with the desired numerical return value. Within a script you can access the returned value of the last process run from the script with the $? special parameter. Because that special parameter changes with each new process, be sure to save it if it will be needed later.

Branching

Here is the syntax for shellscript "if" statements:

if command; then
  statement 1
  statement 2
  ...
fi

• There are no brackets. Brackets are really a substitute for the test executable. The test executable evaluates conditions and file properties. Life is simpler if you just use the test executable. When reading code with brackets in if statements, substitute the word test in your mind.
• The if statement operates on an executable command, not on a condition.
• If the executable returns 0, the statements in the if statement are execuated. CAREFUL! Note how different this is from C, where a true condition triggers an if statement.
• The command being tested must end with a semicolon followed by the word then. The semicolon serves to separate the statement comprised of the if and associated command from the statement comprised of the then. Note there exists an alternative syntax in which the keyword then goes on the next line, in which case the semicolon is no longer necessary in order to separate the two:

if command   then   statements   fi

• However, the preceding syntax is so incredibly ugly it will be discussed no further.
• Shellscripts are not indentation sensitive. They can be, however, space sensitive, especially when using those lame bracket constructs.
• The statements to be executed when the command returns zero are delineated on top by the then keyword, and on the bottom by the fi ("if" backwords) keyword. Even when else and elif clauses are included, there is still only one fi in a non-nested complex if statement.

The following is a very simple if construct that tells you whether or not the first argument was the letter s.
 

if test "$1" = "s"; then   echo "Arg 1 was s."; fi

In the preceding, you can see how the test executable evaluated the condition "$1" = ""s" and returns 0 if that statement is true.

Shellscripts give you full "if, else if, else" capabilities without obnoxious nesting. The following example tests arg1 for either s, t, or something else:
 

if test "$1" = "s"; then   echo "Arg 1 was s." elif test "$1" = "t"; then   echo "Arg 1 was t." else   echo "Arg 1 was neither s nor t." fi

In the preceding, note that there is only one fi keyword. That fi keyword ends the entire if elif else statement. There's no necessity for nesting in an if elif else construct. However, note that the elif statement requires a command, semicolon, and then keyword just like if. It does not require its own fi terminator.

Although nesting isn't necessary for if elif else constructs, in some cases it's desirable. That's not a problem -- it is done with fi keywords. The following evaluates arg2 if arg1 is "s", otherwise not. It nests an if elif else inside the first if statement:

if test "$1" = "s"; then   echo "Arg 1 was s."   if test "$2" = "1"; then     echo "arg2 was 1"   elif test "$2" = "2"; then     echo "arg2 was 2"   else     echo "arg2 was neither 1 nor 2"   fi elif test "$1" = "t"; then   echo "Arg 1 was t." else   echo "Arg 1 was neither s nor t." fi

In the preceding, notice the fi keyword at the end of the nested if elif else, just before the unindented elif . It's that fi that does the nesting.

The Test Command

Relational Operators for test command

if test "$1" = "mystring"; then
  mycommand
fi

Boolean Operators for test command

if test "$1" = "one" -a "$2" = "two"; then
  echo "TRUE"
fi

File Operators for test command

if test -d /home/slitt/umenu; then
  echo DIRECTORY
fi

Case Statements

case "$1" in
  (y|Y|1|yes|Yes|YES)
    echo YES
    ;;
  (n|N|0|no|No|NO)
    echo NO
    ;;
  (*)
    echo Maybe
    ;;
esac

The variable or expression to be matched appears immediately to the right of the case keyword. Each list of matches is in parentheses. Between the closing paren and a line with a double semicolon are all the statements to be executed if there's a match between the expression on the case line and any of the items in the parentheses. The equivalent of "default" in a C switch statement is achieved by making the last choice be an asterisk, which matches anything. The entire case structure is bottom delineated by keyword esac, which is case spelled backwards.

Note that case statements are often used with just one choice. That's because they directly evaluate a match without using the test command or other commands, and because they can match against a list of possibilities. If you've ever wished if statements could be tested against a list of strings, use case.

While maintaining code, you may find that the opening parentheses are omitted. That works, but the bash man page says to use opening and closing parentheses. Also, I think it looks more satisfying than having unmatched parens.

Short Circuit Logic

test $# -eq 0 || echo 1>&2 Information: Command takes no arguments

The preceding is an example of short circuit logic. If the first part is true, there's no need to "evaluate" the second part, so it's not done. If the first part is false, the truth of the entire statement depends on the truth of the second part, so it's "evaluated", which in this case prints an informational message. Note that the same effect can be had with negative logic:

test $# -ne 0 && echo 1>&2 Information: Command takes no arguments

In the preceding, the first clause produces a true, but to make the entire statement true requires truth in both clauses, so the second clause is "evaluated", printing the message.

Short circuit logic is a great shorthand notation when there's exactly one statement to be done upon satisfaction of the condition. Typically, that single statement is a function call (shellscript subroutines are discussed later in this document). However, the ease and conciseness of this form breaks down rapidly when more than one statements must be done if the condition is satisfied. For instance, suppose that you want to print a message AND exit if there are command line arguments. Here's the syntax:
 

test $# -eq 0 || { echo 1>&2 "Error: Command takes no arguments";exit;  }

Notice that braces are used to make both the echo command and the exit command into a single command that is the second clause of the or statement. Without the braces, the exit command would be done regardless of the condition. Notice that the braces must be separated from the commands by a space. That non-obvious shellscriptism is truly ugly, and very hard to troubleshoot when neglected. For all these reasons, short circuit logic is best employed only when there's a single statement to be executed on satisfaction. Otherwise, it's best to use the standard if elif else, or possibly the case statements.

Bracket notation is often used on the condition in short circuit logic. Always remember that the brackets are shorthand for the test executable, and therefore must be separated from the text by space. I recommend that on new code you write, you simply use the test executable.

User Input

echo -n "Please type your name==>"
read namevar
echo "Your name is $namevar!"

This has no input validation. Validation can be had by placing the preceding in a loop that exits on valid input, and re-enquires on bad input. Also, the preceding snippet can be a security hazzard. The most obvious case is if the user is allowed to type in a command (typically in a CGI shellscript).

To restrict the user to a set of choices, use the select keyword, which constructs a sort of poor man's menu. The following is an incredibly simple example which gives the user three choices -- a directory listing of *.sh, a directory listing of *.c, or terminate the select statement by executing a break statement:
 

select choice in "ls *.sh" "ls *.c" "break";do    test "$choice" = "" || $choice; done echo "Dropped through loop"

The test for a blank $choice variable prevents execution of an invalid user choice. Invalid choices assign a NULL to the select variable ($choice in this case).

The preceding throws up a menu that looks like this:
 

[slitt@mydesk slitt]$ ./jj 1) ls *.sh 2) ls *.c 3) break #?

Each number corresponds in to a list element after the in keyword. Note that the $choice variable is filled by the string corresponding to the choice, not the number the user typed. Note also that if the user types anything not corresponding to one of the choices, the $choice variable is NULL.

Most real world menus with error messages and the like include a case statement inside the select construct. This is also vital if the choices are not commands.

Looping

Looping Through Lists

for i in *.sh; do
  echo "Shellscript $i"
done 

The preceding code lists all files in the current directory ending in .sh, and assigns variable i to each one sequentially. Have you ever wondered how to do a wildcard rename in UNIX? You do it with a script like the preceding. Of course the script would have a mv statement instead of an echo.

This gets really handy when you need a multiple file "rename" command. UNIX has no "rename" command, because any "rename" command at best guess what you meant. The following is a one liner to rename files starting with a yymmdd date format in the 90's, and prepending 19 onto the front to make them accurate yyyymmdd:
 

$ for i in 9*.html; do mv $i 19$i; done

shopt -s nullglob

Lists needn't be just filenames. Any space delimited sequence of strings will do. The following makes a list of all the vowels, and echos them:
 

for i in a e i o u; do   echo "Vowel $i" done

Whole words can be used as a list and looped through, as shown in the presidential lister that follows:
 

for i in Reagan Bush Clinton; do     echo "President $i" done

Any space delimited string can be used as a list. Not only that, the list can be modified by changing the space delimited string. The following code snippet places Carter, Reagan, Bush and Clinton in a list of presidents, and then belatedly adds Ford to the front of the list. Then a for statement loops through the presidents:
 

presidents="Carter Reagan Bush Clinton" presidents="Ford $presidents" for i in $presidents; do     echo "President $i" done

The following is a recursive directory lister that loops through all the file objects in a directory, and calls other invocations of itself for subdirectories. The result is a list of all regular files in a tree:
 

directory=$1 for filename in $directory/*; do    if test -d $filename; then      ### -d in test command returns 0 on directory file objects       ./showfiles.sh $filename     ###Recurse into another showfiles.sh    elif test -f $filename; then    ### -f in test command returns 0 on regular files       echo $filename    fi done

Life's not always that simple. Sometimes there's no list to conveniently loop through. In such a case you use a while loop. The following while loop prints numbers from 1 to 4:
 

let repeats=4 let iteration=1 while test $iteration -le $repeats; do    echo "Iteration number $iteration"    let iteration+=1 done

Note that once again, the while loop operates on a commmand, not a condition. Therefore you must use the test executable to convert the condition into a return value.

Generally speaking, you use the let keyword when you want to make the variable a number rather than a string, and/or when you want to do arithmetic on the number.

There's also an until keyword that's identical to while except that it reverses the sense of the condition. In other words, it repeats UNTIL the command being tested returns 0. The following produces output identical to the preceding code snippet, but it uses until instead of while:
 

let repeats=4 let iteration=1 until test $iteration -gt $repeats; do    echo "Iteration number $iteration"    let iteration+=1 done

 

Subroutines

function fibonacci()
  {
  local newnumber
  let newnumber=$1+$2
  echo $newnumber
  return $newnumber
  }

let maximum=100
let n1=0
let n2=1
while test $n2 -le $maximum; do
  fibonacci $n1 $n2
  let temp=$?
  let n1=$n2
  let n2=$temp
done           

In the preceding example, notice that the function is declared with the word function, the name, and a set of *empty* parentheses. Unlike C and the new Perl syntax, subroutine arguments are accessed as $1, $2... and the number of arguments as $#. The statements of the function are between braces. Because braces in shellscripts are space dependent, I believe the best syntax is to have each brace on its own line. Other syntaxes work -- you can experiment. Notice the keyword local. That declares variable newnumber as local to function fibonacci in scope. Otherwise it would be a global variable, even though it's declared inside the function. Global variables are the kiss of death for modularity.

Notice that the function uses its return statement to return a value. If you do not have a return statement, the return value is the return value of the last executable statement executed in the function. That being somewhat arbitrary, it's a good idea to explicitly specify a return value in any function whose return value is important.

Unlike C, Perl and the like, the return value cannot immediately be assigned to a variable. In other words, the following is a no-no:

temp=fibonacci $n1 $n2

File I/O

echo $mystring > myfile.txt

You can get better performance out of the echo statement by collecting large amounts of text in the variable (including newline characters), and then writing the huge text collection to the bottom of the file. Unfortunately, that takes lots of memory, probably coming off the stack.

Of course, you can simply echo to stdout (echo statement without redirection), and then have the calling process redirect your script's stdout to a file. That's a great alternative if practical. But sometimes you just have to byte the bullet and have your script write directly to a file. Here's what you do:
 

exec 44>test.txt let index=0 while test $index -lt 10; do   echo "This is line $index" >&44   let index+=1 done exec 44>-

The first exec statement opens a file on file descriptor 44. The echo statement in the loop writes to that continously open file descriptor. The exec at the end closes the file descriptor.

The problem is that often you need to use a variable for the file descriptor number, and doing so totally messes up the preceding syntax (trust me on this, I tried everything). So to use variables instead of hard coded numbers and strings, you use the eval command as follows:
 

filename=test.txt let filedescriptor=44 let maxindex=10 eval "exec $filedescriptor>$filename" let index=0 while test $index -lt $maxindex; do   echo "This is line $index" >&$filedescriptor   let index+=1 done eval "exec $filedescriptor>-"

Even this does not yield good performance, because every call to echo and every call to test must spawn a program, a very slow task. As a result, the same algorithm in C is over 100 times faster.

Regular Expressions

Shellscripts have filename type wildcards * and ?, which expand the same as in filenames. It is possible to use those as a "poor man's regular expression".

String Manipulation

The following shows a code snippit to indent relative to arg1. The second statement creates the longest possible indent. The third line prints a substring of $indents*2 spaces, without printing a newline (the -n suppresses the newline).

indents=$1 indentstring="                                                             " echo -n "${indentstring:0:$indents*2}" echo "Hello"

The syntax ${stringvar:offset:length} is a "substring" type function that works very well. To prepend a string to a string variable, simply put both inside a single set of doublequotes as follows:
 

mystring=xyz newstring="abc$mystring" ###Following line prints "abcxyz"### echo "$newstring"

However, if you append literal "abc" to the variable, the shell will interpret it as a variable called $mystringabc, clearly not what you want. So you place braces around the variable name (but not the dollar sign), to isolate the variable name:
 

mystring=xyz newstring="${mystring}abc" ###Following line prints "xyzabc"### echo "$newstring"

You can get the length of a string variable with the ${#parameter} syntax:
 

mystring="1234" let stringlength=${#mystring} echo "String is $mystring, whose length is $stringlength"

A string variable can be subjected to search and replace with the following syntax:
 

mystring="I say replace me before it's too late to replace me!" mystring=${mystring//replace me/replaced} echo $mystring

A couple facts about the preceding syntax. The ${parameter//pattern/replacement} syntax does not, in and of itself, change the parameter string. The reason $mystring changed is because I assigned the value of ${mystring//replace me/replaced} back to $mystring. It could have just as easily been assigned to something else, leaving $mystring untouched. Also, note that the preceding syntax replaces all instances of the pattern text. To replace just the first instance, use a single slash instead of the double slash.

Sometimes you need to walk through a string finding occurrences of a word. Check out this script, which uses the word "delimit" to delimit parts of the string:
 

mystring="ReagandelimitBush SeniordelimitClinton" name="primingvalue" while test ${#name} -gt 0; do    xstring=${mystring#*delimit}   # Take shortest *delimit off front of $mystring    name=${mystring%$xstring}      # Take shortest $xstring off back of $mystring    name=${name%delimit}           # Take shortest delimit off back of $name    if test ${#name} -gt 0; then       echo $name    else       echo $mystring    fi    mystring=$xstring done

If this script is called walkthrough.sh, here is the result:
 

[slitt@mydesk slitt]$ ./walkthrough.sh Reagan Bush Senior Clinton [slitt@mydesk slitt]$

Here's a statement by statement description:

mystring="ReagandelimitBush SeniordelimitClinton"

name="primingvalue"

while test ${#name} -gt 0; do

   xstring=${mystring#*delimit}

"*delimit" means any string followed by string "delimit". The fact that there is only a single pound sign (hash mark) means expand the search text to the shortest such possible string. If there had been two pound signs it would have expanded the search text to include everything up to the final "delimit" string, which is not what we want. The pound sign operator used in this way returns a substring, to variable $xstring, of the contents of $mystring *after* the end of the first "delimit" string. In other words, on the first loop through, it removes "Reagandelimit" and returns the rest. Because we will later be assigning $xstring to $mystring , we will iteratively walk through the string.

Notice that this basically returns *everything but* what we really want, which is the name at the front of the string. The next few statements take care of that.

   name=${mystring%$xstring}

So after the execution of this statement, $name contains the name we want with a trailing "delimit" string, which must be gotten rid of. Note, however, that if no more "delimit" strings exist, this construct returns an empty string (or NULL, or whatever) to the $name variable. That is how the loop is exited.

   name=${name%delimit}

   if test ${#name} -gt 0; then
      echo $name
   else
      echo $mystring
   fi

   mystring=$xstring

done

The preceding example is pretty ugly, and one excellent example why Perl and Python (and C and Java) are often chosen over shellscripts. Nevertheless, it goes to show that if you're willing to get down and dirty, shellscripts often have the power you need.

The next sample is the walkthrough.sh shellscript rewritten. It directly grabs the first name on the list using %% to delete all before the first delimit, thus eliminating the extra step to remove delimit from the end of the name. It also detects the last record directly by comparing the length of $mystring to its length on the previous iteration, thereby removing the need for the if statement.
 

mystring="ReagandelimitBush SeniordelimitClinton" name="primingvalue" let lastlength=99999              # Longer than any anticipated $mystring while test ${#mystring} -lt $lastlength; do    let lastlength=${#mystring}    # Prepare break logic    name=${mystring%%delimit*}     # Get everything before first delimit    echo $name                     # Output this name    mystring=${mystring#*delimit}  # Take shortest *delimit off front of $mystring done

Parsing Filenames and Mass Renames

#!/bin/bash

shopt -s nullglob
for i in *.JPG; do
	base=$(basename $i .JPG)
	newname=${base}.jpg
	echo $i "   " $base "   " $newname
done

#!/bin/bash

shopt -s nullglob
for i in /home/slitt/*; do
	echo $i
	base_name=$(basename $i)
	dir=${i//$base_name/}
	no_ext=$(echo $base_name | sed "s/\.[^\.]*$//")
	ext=${base_name//$no_ext/}
	ext=${ext//./}
	echo Full name = $i
	echo Base name = $base_name
	echo Dir       = $dir
	echo No Ext    = $no_ext
	echo Ext       = $ext
	echo
	echo
done

Quasi Object Programming

 [ Troubleshooters.com | Code Corner | Email Steve Litt ]

Copyright (C)2000-2002 by Steve Litt -- Legal