Troubleshooters.Com®, Linux Library Present:
Execline 101
Copyright © 2019 by Steve Litt
See the Troubleshooters.Com Bookstore.
CONTENTS
Execline is a language designed from the ground up to perform fairly simple scripts. Execline scripts are often used as an alternative to shellscripts (/bin/sh or /bin/bash), especially but not exclusively when used in run scripts of process supervisors.
Execline scripts distinguishes themselves from /bin/bash or /bin/sh on smallish, straightforward scripts, by using less RAM, not staying resident in memory while execution proceeds, and having a more consistent syntax. Like /bin/bash and /bin/sh, execline delivers unacceptable performance when executing tight loops with over 100 iterations. Use Perl/Python/Ruby/Lua/C, or other "real languages", in those situations.
Likewise, if the logic gets too hairy, or if you find yourself needing data structures, use "real languages". But for relatively straightforward stuff, execline is in my opinion a more straightforward syntax than Bash or /bin/sh, and might save you some RAM.
Execline is a different breed. It's not natively procedural, but instead tends to use the first word of the command line as a program, and use the rest of the words as arguments to that program. It's important to realize that execline uses the exec function a lot.
NOTE:
It actually uses the execve function, but I'm speaking of it as a concept, so I use the word "exec".
The exec function works kinda sorta like the following, as a gross generalization not applicable to real life:
You can have a whole chain of execs, where A execs B, which execs C, which execs D, and when all is said and done, D is running with the PID of A. Note that A, B, C and D can do some work before execing their successor.
This type of behavior is used quite a bit in execline. Some people call it "chaining". Some call it "Bernstein Chaining". Whatever it's called, the principle is that each program execs the next, giving up all its RAMin the process, which in many cases saves on RAM.
It would be counterproductive to discuss such chaining further here: Instead use this document as a tutorial. Just be aware that this is not procedural programming, but instead it's based on chaining.
WARNING!
Many portions of this document describe program processes with their use of exec and fork, etc. These descriptions are approximations for the purpose of introductory understanding: They are not completely accurate. For the absolutely accurate story, you'll need to look at the source code at https://github.com/skarnet/skalibs and https://github.com/skarnet/execline, and if you want to go really deep,the posix source code implementing the functions in /usr/include/spawn.h.
In addition, there are elements of the execlineb program that I don't even touch: Parsing, substitution, and argv assembly. I don't think you need intimate knowledge of these things to write casual execline code. Beware, however, if you're speaking to an execline/execlineb expert, you'll rightfully be called wrong for restating some of these descriptions, even though these descriptions helped you learn to code execline.
Here's the simplest Hello World execline program:
#!/bin/execlineb echo Hello World!
Permission it executable, run it, and you'll see exactly what you expect. This is the simplest possible execline program: It's deceptively simple, as you'll soon see.
If you look at the shebang line (the line starting with #!), it references execlineb, not execline. The explanation follows:
What happened in this program is that the execline program exec'ed the echo program with the arguments Hello and World!.
Let's try to add a second line of output, the obvious way, to this Hello World program, and watch it fail:
#!/bin/execlineb echo Hello World! echo 2
The preceding should print "Hello World!" on one line, and "2" on the next line, right? Well, ummm, here's what really happens:
[slitt@mydesk test]$ ./hello.e
Hello World! echo 2
[slitt@mydesk test]$
In the preceding output, we see that not only did it print just one line, but the second instance of the word "echo" was just taken as one more piece of data to be printed. The reasons are as follows:
So what happened is that execline exec'ed echo with arguments Hello, World, echo and 2. Which explains the exact results obtained.
The next section lays out the right way to print these two lines.
The following execline program prints the two lines:
#!/bin/execlineb
foreground { echo Hello World! }
echo 2
The preceding code produces the following output:
[slitt@mydesk test]$ ./hello.e
Hello World!
2
[slitt@mydesk test]$
The foreground command is actually a special executable program that ships with execlineb. When run within execlineb, foreground runs exactly two programs, one at a time. It forks the first program, and when that program exits, it execs into the second program. The following is a stepwise description of what the foreground program does:
In the execline language, those curly braces are not punctuation, they're separate words. In the execline language words are separated by whitespace, any kind of whitespace: One space, two spaces, one or more tabs, or one or more newlines: All these kinds of space are the same to execline: They separate words.
The curly brace words are used to delineate a block. The typical use of a block is to group a program with all its command line arguments.
Let's say you didn't use braces to define a block:
foreground a b c d e f
In the preceding, you know for a fact that a is a program to be executed. But what is b? It could be the second program, or it could be an argument to the first. The exact same thing could be said for c d and e. In order to discern which args go with the first program, you need a block.
Let's briefly discuss formatting, using shorter echo commands. I formatted mine as:
#!/bin/execlineb
foreground { echo 1 }
echo 2
Because newlines are just space in execline, the preceding could be written:
#!/bin/execlineb
foreground { echo 1 } echo 2
In fact, it is written that way in much existing documentation. For reasons pointed out in the next section, I prefer putting a newline in the execline source code between the first and second program to be run by foreground.
The preceding section illustrated how to use foreground to run two programs consecutively. But how do you run three programs consecutively? It turns out quite simple if you format it right. The following is an example, this time using the short echo commands:
#!/bin/execlineb
foreground { echo 1 }
foreground { echo 2 }
echo 3
The preceding code outputs the following:
[slitt@mydesk test]$ ./test.e 1 2 3 [slitt@mydesk test]$
This code could have been formatted on one line:
#!/bin/execlineb
foreground { echo 1 } foreground { echo 2 } echo 3
Your mileage may vary, but to me, the 1 line approach makes a mistake with quote inclusion more likely. I prefer the line by line:
#!/bin/execlineb
foreground { echo 1 }
foreground { echo 2 }
echo 3
The preceding program runs as follows: Execline forks foreground, which forks echo 1, after the termination of which it execs the second foreground, which forks echo 2, and after the termination of echo 2, the second foreground execs echo 3. To really study what forks what, what execs what, and what stays in memory and what doesn't, I suggest studying the following program in the ps axjf | less command:
#!/bin/execlineb
foreground { gnumeric }
inkscape
If you don't have either of the preceding two GUI programs, use almost any GUI program. Don't use gvim, because it double-forks itself and instantly returns control to whatever called it.
First try the following:
#!/bin/execlineb define myvar Hello echo $myvar
When you run the preceding, you get exactly what you expect. The define statement assigns "Hello" to name myvar. Very much like Bash, you use a dollar sign when using a variable, but not when assigning a value to it.
The define command/program works as follows:
Now try the following:
#!/bin/execlineb define myvar Hello World! echo $myvar
The preceding errors out, producing the following output:
[slitt@mydesk ~]$ ./junk.e
define: fatal: unable to exec World!: No such file or directory
[slitt@mydesk ~]$
The explanation of the preceding error is that define assigned "Hello" to myvar, and then ran the next word, "World!".
So try to fix it as follows:
#!/bin/execlineb
define myvar { Hello World! }
echo $myvar
Running the preceding produces the same error message. With both "Hello" and "World!" in a group, one would think it would be taken as a single element, but for some reason that doesn't happen.
The correct technique is as follows:
#!/bin/execlineb define myvar "Hello World!" echo $myvar
The preceding produces a "Hello World!" output.
Now watch the following fail:
#!/bin/execlineb define myvar "Hello World!" echo "$myvar to everyone!"
The preceding code produces the following output:
[slitt@mydesk ~]$ ./test.e
$myvar to everyone!
[slitt@mydesk ~]$
Fix it by putting curly braces directly around the variable name, as follows:
#!/bin/execlineb
define myvar "Hello World!"
echo "${myvar} to everyone!"
The preceding code produces the desired output:
[slitt@mydesk ~]$ ./test.e
Hello World! to everyone!
[slitt@mydesk ~]$
Whenever you want to put a variable substitution, like $myvar or $1 in a doublequoted string, it's important to protect that variable substitution with curly braces, like ${myvar} or ${1}. Otherwise the output will contain only the variable name preceded by a dollar sign.
It's important to differentiate, in your mind, the difference between using curly braces to protect a variable name, in which case the braces are flush up against the name, as opposed to use of curly braces to form a group, in which case the curly braces are standalone words, surrounded on both sides by space.
Remember, when a variable prints as its name rather than its value, the two things to check are:
As far as I know, it's never a problem to brace-protect a variable, so it might be a good idea to always protect as a habit.
Create the following test.e:
#!/bin/execlineb ls /etc/f*
Look what happens when you run it:
[slitt@mydesk test]$ ./test.e
ls: cannot access '/etc/f*': No such file or directory
[slitt@mydesk test]$
It turns out that execline cannot directly handle the stars and question marks used in file globbing. Execline itself could have found /etc/fstab, but not /etc/f* To do file globbing within execline, you need to use the elglob program.
The preceding section demonstrated that yoou can't directly use file wildcards in execline. So you need to use the elglob program, as follows:
#!/bin/execlineb
elglob fil /etc/f* echo ${fil}
Here's the explanation. elglob is an execline program, as is foreground. elglob requires a variable name as its first arg, the file pattern as its second arg, and a command as its third arg:
I won't show the output because it all comes on one line, but the output is every file in /etc that begins with f, separated by spaces.
An equivalent but easier to handle format for the preceding program is the following:
#!/bin/execlineb
elglob fil /etc/f*
echo ${fil}
Something handy is to put the variable value in quotes, and put a newline at the end:
#!/bin/execlineb
#ls /etc/a*
elglob fil /etc/f*
echo "${fil}\n"
Notice the only change in the immediately preceding version was putting a newline behind the variable value, and putting the variable and value combination in quotes. This changes the output from a one liner to the following:
[slitt@mydesk test]$ ./test.e /etc/freepats /etc/fppkg /etc/fstab.bup /etc/fstab.org /etc/fonts /etc/fstab /etc/fpc.cfg /etc/fail2ban /etc/fuse.conf /etc/fppkg.cfg [slitt@mydesk test]$
Everything seems cool except for the single space prepended to each but the first line, and the extra newline at the end of the list. Ways to fix that are described later in this document, but for the time being, this is a great way to get quick info.
One other problem is what happens if no matching files are found. My /etc has no files beginning with "y", so let's change the pattern to /etc/y*:
#!/bin/execlineb
#ls /etc/a*
elglob fil /etc/y*
echo "${fil}\n"
The output follows:
[slitt@mydesk test]$ ./test.e /etc/y* [slitt@mydesk test]$
The preceding is just plain inaccurate. I have no file in /etc named y*. To fix this problem, you need to add -0 to the elglob command, as shown in the following version:
#!/bin/execlineb
elglob -0 fil /etc/y*
echo "${fil}\n"
The preceding does the right thing whether there are any matches or not.
There are many other command line options for elglob. You can find those in the official documentation at https://skarnet.org/software/execline/.
This section barely scratches the surface of loops. It's a loop Hello World, if you will. Some things, such as the necessity of importas inside a loop, aren't readily evident in most execline documentation.
Create the following test.e:
#!/bin/execlineb
forx myvar { a b c d e }
importas myvar_inside myvar
echo "${myvar_inside}"
The preceding code produces the following output:
[slitt@mydesk test]$ ./test.e a b c d e [slitt@mydesk test]$
Re-showing the code, so it can be discussed line by line...
#!/bin/execlineb
forx myvar { a b c d e }
importas myvar_inside myvar
echo "${myvar_inside}"
NOTE:
The indentation of the importas and echo lines isn't necessary for function: I did it to make it clear that those two lines were exec'ed by the forx line on each iteration.
First, create the following test jig shellscript, called testjig.sh, to display the executable path, as one directory per line, and then all arguments sent to it:
#!/bin/sh echo ================== echo $PATH | sed -e"s/:/:\\n/g" echo ================== echo $@ echo ==================
Next, create the following path prepender execline script, called prepath.e, whose arg1 is the directory or series of directories to put on the front of the path, and whose arg2 is the program for the prepender to execute (after prepending the path, and all additional arguments are arguments for the program executed by the prepender:
#!/bin/execlineb -s2
importas OLDPATH PATH
export PATH "${1}:${OLDPATH}"
importas PATH PATH
${2} ${@}
Examine the preceding execline script:
The execline language has four command programs for branching:
This section discusses the first three. Start with an example showcasing if:
The best use of if is when you want the script to continue only if the test is true. Here's a basic example:
#!/bin/execlineb -s1
if { test $1 -gt 10 }
echo ${1} is Greater than ten
Run the preceding with an argument of 4, then an argument of 14, and notice it prints on the latter but not on the former. If the program run as a test delivers up something other than 0, all execution of the script stops. Observe also that with the use of foreground and other things, this script could be modified to do five or ten more things after the program being tested returns a zero.
Best use of ifelse is when you want to do one thing if true, another thing if false, but either way, execution ends there. Nothing following the ifelse is executed. Or if you want the script to stop after executing the program to be run if true, but to continue if false. A simple example follows:
#!/bin/execlineb -s1
ifelse { test $1 -gt 10 }
{ echo ${1} is Greater than ten }
echo ${1} is Not greater than ten
In the preceding code, the program to be tested is still the first one, but now it is followed by two programs: One in a brace block to be executed if the first one returned 0, and one without a brace block to be executed if the first one returned something other than 0. Try it with command line argument 4, and then 14, and see what happens.
Use ifthenelse when you want execution to continue beyond the test and whatever programs the test runs. One typical use is when you want to set some variables whose values depend on the test, but then the rest of the script will use those variables. The following is an example:
#!/bin/execlineb -s1
ifthenelse { test $1 -gt 10 }
{ echo ${1} is Greater than ten }
{ echo ${1} is Not greater than ten }
echo Either way, life is good.
Once again, try the preceding first with an arg of 4, then an arg of 14, and you'll get the picture.
Just for fun, check this out:
#!/bin/execlineb -s1
ifelse { test $1 -gt 10 }
{ echo ${1} is Greater than ten }
ifthenelse { test $1 -eq 10 }
{ echo ${1} is Equal to ten }
{ echo ${1} is Less than ten }
echo Whichever way, life is good.
This subsection starts with an immitation case statement built with ifthenelse commands, and then presents a possibly simpler version created with one foreground command and a bunch of ifelse commands. Let's start with the ifthenelse version. Just as a review, ifthenelse looks like the following:
ifthenelse { testpgm } { truepgm } { falsepgm }
Which, because newline is just another space in execline, is equivalent to:
ifthenelse { testpgm }
{ truepgm }
{ falsepgm }
If testpgm returns 0, that's considered true. If it returns a different number that's considered false. Several ifthenelse commands can be nested to produce a pseudo case statement that's really quite convenient and readable. See the following example:
#!/bin/execlineb -s1
ifthenelse -X { test $1 -eq 1 }
{ echo ${1} equals 1 }
{ ifthenelse -X { test $1 -eq 2 }
{ echo ${1} equals 2 }
{ ifthenelse -X { test $1 -eq 3 }
{ echo ${1} equals 3 }
{ ifthenelse -X { test $1 -lt 1 }
{ echo ${1} less than 1 }
{ ifthenelse -X { test $1 -gt 3 }
{ echo ${1} greater than 3 }
{ echo ${1} internal error } # ending default
} } } }
echo Continuing on with rest of program.
The preceding code produces the following first-line results, depending on argument:
| arg1 | 1st line |
| 0 | 0 less than 1 |
| 1 | 1 equals 1 |
| 2 | 2 equals 2 |
| 3 | 3 equals 3 |
| 4 | 4 greater than 3 |
| nan | internal error |
The second line is always "Continuing on with rest of program.".
Keep the following tips in mind to make these case statements easy:
An arguably easier way to make a pseudo case statement is to use a foreground command with a bunch of ifelse commands. Such a construction relieves one of counting braces. The following source code produces the exact same result as the preceding execline program:
#!/bin/execlineb -s1
foreground {
ifelse -X { test $1 -eq 1 } { echo ${1} equals 1 }
ifelse -X { test $1 -eq 2 } { echo ${1} equals 2 }
ifelse -X { test $1 -eq 3 } { echo ${1} equals 3 }
ifelse -X { test $1 -lt 1 } { echo ${1} less than 1 }
ifelse -X { test $1 -gt 3 } { echo ${1} greater than 3 }
echo ${1} internal error # ending default
} # Foreground ending brace
echo Continuing on with rest of program.
Notice you don't have to insert a row of ending braces: You just put one ending brace after the case's default. The preceding looks great if all the "then clauses" are short. If some are long enough to require multiple lines, the preceding can be reformatted as follows:
#!/bin/execlineb -s1
foreground {
ifelse -X { test $1 -eq 1 }
{
echo ${1} equals 1
}
ifelse -X { test $1 -eq 2 }
{
echo ${1} equals 2
}
ifelse -X { test $1 -eq 3 }
{
echo ${1} equals 3
}
ifelse -X { test $1 -lt 1 }
{
echo ${1} less than 1
}
ifelse -X { test $1 -gt 3 }
{
echo ${1} greater than 3
}
echo ${1} internal error # ending default
} # Foreground ending brace
echo Continuing on with rest of program.
Just like /bin/sh, large scale numerical looping is where you run into time trouble with execline. Consider the following numloop.e:
#!/bin/execlineb -s1
forbacktickx i { seq $1 }
importas i i
echo $i
In the preceding code, the -s1 enables the capture of the first command line argument, which of course is the number to loop until. The importas i i enables the use of $1 within the forbacktickx construct. This code enables time testing, as follows:
[slitt@mydesk ~]$ time ./numloop.e 10 > /dev/null real 0m0.056s user 0m0.013s sys 0m0.024s [slitt@mydesk ~]$ time ./numloop.e 100 > /dev/null real 0m0.417s user 0m0.082s sys 0m0.233s [slitt@mydesk ~]$ time ./numloop.e 1000 > /dev/null real 0m4.797s user 0m0.888s sys 0m2.338s [slitt@mydesk ~]$ time ./numloop.e 10000 > /dev/null real 0m47.838s user 0m8.159s sys 0m23.914s [slitt@mydesk ~]$ time seq 10000 > /dev/null real 0m0.002s user 0m0.000s sys 0m0.001s [slitt@mydesk ~]$
The preceding time tests are telling. The time execline takes to complete the task seems pretty much proportional to how many numbers you loop through. Looping 100 times takes a half second: No problem. Looping 1000 times takes almost 5 seconds, which is acceptable if it doesn't happen often and if a prompt warns the user to expect a delay. At this point you might consider trading execline for a language like Python, Lua, Ruby or Perl. Looping 10000 times takes almost 50 seconds: Completely unacceptable: Use a different language.
If you don't need file globbing, execline is trivial:
#!/bin/execlineb ls /etc
As mentioned before, when globbing is necessary, the elglob program is necessary. The following is an elglob enabled script that's sophisticated enough to do work on each file found:
#!/bin/execlineb
elglob -0 files /etc/f*
forx file { ${files} }
importas file file
echo "File: ${file}"
The preceding code produces the following output:
[slitt@mydesk test]$ ./test.e
File: /etc/freepats
File: /etc/fppkg
File: /etc/fstab.bup
File: /etc/fstab.org
File: /etc/fonts
File: /etc/fstab
File: /etc/fpc.cfg
File: /etc/fail2ban
File: /etc/fuse.conf
File: /etc/fppkg.cfg
[slitt@mydesk test]$
The following works in bash type shellscripts, but not in execline:
ls -1t | head -n5
Instead, you need to use execline's pipeline command program. The following execline script prints the top 20 biggest files in the /etc directory (not tree):
#!/usr/bin/execlineb
pipeline { ls -l /etc } # all entries in /etc
pipeline { grep -v "^d" } # no directories
pipeline { sed -re "s|\\s+| |g" } # turn multiple space
# into single
pipeline { cut -d " " -f 5,9- } # Field 5, and 9 and above
# fld 5 is size, 9- fname
pipeline { sort -rn } # Sort numerically reverse
pipeline { head -n 20 } # Only the top 20
cat -n # Add rank numbers on left
By the way, that double backslash in the sed filter is necessary so that sed reads it as a single backslash. The "9-" in the cut filter assures whole filenames printed if somebody unwisely puts spaces in filenames.
Perhaps you need to do other tasks after all this pipelining. You can use a foreground command program to accomplish this, making the whole bunch of pipelines a single group, as follows:
#!/usr/bin/execlineb
foreground {
pipeline { ls -l /etc } # all files in /etc
pipeline { grep -v "^d" } # no directories
pipeline { sed -re "s|\\s+| |g" } # turn multiple
# space into single
pipeline { cut -d " " -f 5,9- } # Field 5, and 9 and above
pipeline { sort -rn } # Sort numerically reverse
pipeline { head -n 20 } # Only the top 20
cat -n # Add rank numbers on left
}
echo "I finished"
My UMENU2 software is in a directory tree of its own, not on the executable path. My UMENU2 software takes an argument telling it which of myh many menus to display, and it has a -t option that makes it terminate when it runs a program, instead of doing more menuing. The shellscript to run UMENU2, in the terminating mode, on a menu defined by arg1, looks like the following:
#!/bin/sh cd ~/umenu2/prog ./umenu.py -t $1
The execline script that does exactly the same thing, if called from bash or /bin/sh follows:
#!/bin/execlineb -s1
importas HOME HOME
cd ${HOME}/umenu2/prog
./umenu.py -t $1
The reason the execline script is one line longer is that execline has no shorthand for the home directory similar to bash's ~, so it has to import the calling shell's $HOME. And if it's expected to be called from a shell-less situation (boot, for instance), or with a shell substantially different from bash (csh, for instance), then you can't even count on the environment variable $HOME, so it gets a little longer:
#!/bin/execlineb -s1
backtick -n username { whoami }
importas username username
backtick -n myhome { homeof ${username} }
importas myhome myhome
cd ${myhome}/umenu2/prog
./umenu.py -t $1
The reason the preceding execline script is longer than the one before is that one can't count on bash's $HOME environment variable, so it must be deduced by Linux command whoami followed by execline command homeof.
The document you're now reading can take you only so far. Sooner or latre you need more advanced documentation, and that's what this section is for.
If you've worked along with the tutorial formed by this document, you're ready to start writing reasonably simple execline scripts. You'll have an intuitional knowledge of execline, and you'll probably start to think of its syntax as more consistent and less quirky than bash and its brethren.
This document also gives you enough vocabulary and practical execline knowledge to ask questions on the skarnet or s6 mailing list or the #s6 channel on the Freenode IRC server.
What you won't get from this document is the theoretical knowledge necessary to know just how much time and resource you save (or don't) by choosing execline, because you don't truly know how execlineb works under the hood.
Another thing you don't get from this document is the knowledge to speak authoritatively about execline or execlineb. In fact, some of the "facts" in this document are actually a little bit "wrong" if you get down in the weeds, kind of like Newtonian physics is wrong at high speeds. It's a good approximation for certain circumstances.
To take the next step, thoroughly read all the non-source-code links in this document's Pointers to Execline Documentation section. Also, get involved with the mailing list and/or IRC channel. Keep in mind that this IRC channel and mailing list house very knowlegeable people, so read all the material in the links first.
Finally, if you really want to go the extra mile, download the source code at the source links in the Pointers to Execline Documentation section.
Beyond all that, the best thing to do is strategically replace a few short shellscripts with execline scripts, and see how it goes.
[ Training | Troubleshooters.Com | Email Steve Litt ]