Troubleshooters.Com Presents

Linux Productivity Magazine

January 2008

Application Assembly using Small Executables

Copyright (C) 2008 by Steve Litt. All rights reserved. Materials from guest authors copyrighted by them and licensed for perpetual use to Linux Productivity Magazine. All rights reserved to the copyright holder, except for items specifically marked otherwise (certain free software source code, GNU/GPL, etc.). All material herein provided "As-Is". User assumes all risk and responsibility for any outcome.

See also Troubleshooting Techniques of the Successful Technologist
and Rapid Learning: Secret Weapon of the Successful Technologist
by Steve Litt

[ Troubleshooters.Com | Back Issues |Troubleshooting Professional Magazine ]
NoSQL is an extension of the UNIX environment, making available the full power of UNIX during application development and usage.  -- From the Philosophy of NoSQL

CONTENTS

Editor's Desk

By Steve Litt
I'd been doing a long time before it sunk in.  

Early 2004. Needing an app to track billable time, I quickly coded up a Perl program to manage a time log. For simplicity and quick coding, the program received its commands as unmemorable command line arguments. Same for the program that read and reported on the time log. To give the programs an easy user interface, I incorporated them into my UMENU menu. Switching from one project to another became 3 to 6 fully prompted keystrokes.

The actual programs were trivial to write because their total user interface were their arguments. No need for "user friendly" getopt -- it was assumed the programs would be called with the arguments in the right order. Once the programs were written and tested, it was maybe a 20 minute task to incorporate them into UMENU. In 3 or 4 hours I'd written a complete and useable time tracking system.

From that moment forward it was obvious that UMENU could front end just about any command, but I didn't realize the full significance of that fact. More executables were front-ended by UMENU through the years. During those same years I searched for faster application development methodologies, concentrating on Data Centered Programming (DCP).

Meanwhile, the increasing frequency of my business trips made it obvious that sooner or later Troubleshooters.Com needed to be automated in a way that my wife or daughter could fill book orders. Ruby on Rails seemed the quick and easy choice, but the more I saw of it the less I liked it for something I'd have to use myself. Webapps just aren't as productive as thick client apps made for touch typists.

VimOutliner project maintainer Noel Henson occasionally enthused about a development environment called /rdb. /rdb and its free software competitor, NoSQL, stressed using the Unix commands and Unix system of pipes to write quick applications. It made a lot of sense. I read the NoSQL Philosophy page and the /rdb centric book "UNIX Relational Database Management", and got hooked on the idea of assembling applications from small executables.

Because NoSQL is free software, I chose NoSQL and learned all about it. You can set up and populate a NoSQL table in about 5 minutes. It's nothing but an ascii file. It's queried and filtered using tiny executables with names like column, sorttable, indextable, keysearch, getrow, jointable, and updtable. Because the table is just a file, you can also use Unix commands like grep, head and tail.

But NoSQL and /rdb were written by developers for developers. They're outstanding data manipulators, but lack the user interface necessary for use by non-developers. The user must type commands to get results. Since my goal was to have my wife and daughter operate the system, a user interface was needed. If that user interface could be accomplished using small executables, then the NoSQL and /rdb philosophies could be used to implement a true application. These executables would be needed:
I already had the menu executable in the form of UMENU, but writing the picklist and form executables seemed nontrivial, so the project was backburnered.

The Music Player

Mandrake Linux shipped with the Xmms music player. It was a great product and I got used to it. I digitized my vinyl records and play them on Xmms. But starting with Mandriva 2007, Xmms was no longer packaged with the distribution. It still shipped with players Amarok and Rhythmbox, but Amarok is a memory hog, Rhythmbox is complex, over-featureful and confusing. Downloading and trying some other music players brought no joy. An attempt to download and compile Xmms devolved into a brutal match against dependency hell. Crazy as it sounds, creating my own music player began looking plausible.

Of course, developing a graphical music player wasn't plausible. Few of us have the time to do the version dance with WxPython and WxWindows. Perl/Tk is a whole lot of work just to make a music player. And when you really think about it, a music player has only a few functions, none of which require graphics:
Someone on the GoLUG list told me the mplayer executable could be frontended. A day of reading the (not too clear) mplayer documentation yielded slave mode, slave mode commands, and fifo usage. A fifo, sometimes called a named pipe,  is a special file that's written at one end and read at the other -- in this case written by my front end and read by mplayer. Experimentation with a fifo using a command prompt indicated that yes, you can control most of the necessary functions by typing commands into the fifo read by mplayer.

So I created a new menu tree in my UMENU EMDL file, compiled, and bang, mplayer could be started, stopped and controlled from UMENU. In 15 minutes of actual work, not counting the research of course, I'd created a music player. 15 minutes!!!

The new app was far from perfect. The main problem was there was no way to pick the playlist from a list of playlists. Instead, UMENU's prompted argument substitution feature asked for a playlist file, and the user had to type in the full pathname. What a hassle! Worse yet, you can't start mplayer without a playlist (or song, but that's not how I wanted my app to work), so I had to default to a certain playlist, and then go through the hokie procedure to change it. Ughhh!

What was needed was a filepicker, so I wrote one. Written in C, it lists the files in a directory and provides an arrow on the left side that the user can move with the appropriate keystrokes. The user can choose one. If the chosen file is a directory (directories are preceded by a forward slash), then that directory is explored. If the directory is the doubledot directory, the parent directory is explored. The user can either choose a file, or quit, depending on the keystroke he hits.

The true beauty of the filepicker is the incredibly thin interface. The calling program inserts the starting directory into an intermediate file, and then calls the filepicker with the name of the intermediate file as the one and only command line argument. When the filepicker finishes due to the user either selecting or quitting, the filepicker writes the full pathname of the chosen file, and the user's action (did he select or quit), after which control returns to the calling program. Armed with the user's intent, the calling program erases the intermediate file and does the proper thing (for instance, switching to the user's desired playlist).

In order to make the filepicker multiuser and multitasking, the intermediate file must be unique. Unix's mktmp is the perfect way to do this.

Calls to the filepicker were incorporated into the "Load mplayer" and "Change playlist" UMENU commands. Now when the user chooses either of these menu items, the files in the playlist directory are listed so the user can choose the desired playlist, after which that playlist is played. A huge useability improvement.

The Recordpicker

If you look at the list of necessary executables, earlier in this article, you notice that a filepicker isn't one of them. The picker truly necessary is the picklist, or as it's sometimes called, the recordpicker. Luckily, the filepicker already contained about 3/4 of the code necessary to make the recordpicker. I placed the common code in separate files, wrote the recordpicker, tested it, and used it to implement the "Jump to song in current playlist" functionality.

Of course, it was necessary to know the current playlist. mplayer provides a function to get the current playlist, but I couldn't get it to work. So I created a tiny executable whose job was to store and retrieve keyvalue pairs. I wrote the persist executable in Ruby, with the following usage:
[slitt@mydesk ~]$ persist

Usage: persist key=value [file]
 persist key=? [file]
 persist key= [file]

The first syntax stores the value for the key.
The second syntax prints the value on stdout.
The third syntax deletes the key and value.

To retrieve a value from a shellscript, do this:
envvar=$(persist key=? [file])

To retrieve a value from a perl script, do this:
var=`persist key=? [file]`

The key must not contain spaces, nor can
there be spaces immediately before or after the
the equal sign. If the value contains spaces,
the key/value pair must be enclosed in single
or double quotes.

[slitt@mydesk ~]$

It was later pointed out to me that the easier and more Unixly correct way to implement storage of key value pairs would have been to use filenames as the key and file contents as the value. This would also have facilitated storage of binary data, and I might later change persist to do just that. But for the time being, it works perfectly and has the fringe benefit that it provides an easy way to parse equal sign separated keyvalue pairs.

Armed with the persist executable, the program stores the new playlist's filename every time the user starts mplayer or chooses a new playlist. Then, when the user chooses to jump to a different song in the current playlist, persist can deliver the current playlist.

I spent much too much time writing the filepicker and recordpicker. The filepicker has no filetype filter or other filtering on attributes of the filename. The recordpicker has no facilities for multiple choices (tag many choices). But the really great news is, unless tagging multiple choices or screening out certain types of files is needed, these executables can be used, as is, forever.

Pretty Easy to Debug

My new music player, complete with "change playlist" and "jump to song", had a bug. Choosing "jump to song" and choosing the first song caused the player to play the second song instead of the first song. Choosing any other song caused the correct song to be played.

So I added a line to my UMENU setup to write the chosen record number out to a file. This proved that the record number for the first song was 0, and the second song was 1. The recordpicker was setting the record number correctly. At the command line, I wrote echo pt_step 0 > ~/steve.fifo, and verified that it skipped to the next song rather than staying put. mplayer was inconsistent when skipping songs, with 0 being a special case. No problem, in the UMENU setup I placed an if statement to execute the pt_step only if the record number was nonzero.

The outstanding modularity of the recordpicker and mplayer made debugging trivial. I could access most needed information straight from the command prompt. The one line of code needing change was a debugging statement in the UMENU EMDL file. That change was obvious and trivial. The bug took maybe 5 minutes to fix. Contrast that with what you'd need to do in a monolithic program, especially in a compiled language. Just finding the place where the song is changed could take a half hour.

Applications assembled from small executables are pretty easy to debug.

Aftermath

I'm a believer. If I need to write a quick app for use at Troubleshooters.Com, this is the way I'll do it. During the next few months I'll be looking into ways to implement a form executable, which is by far the most challenging executable that will be required. As far as data access, those executables already exist in the form of mysql, psql, or the executables bundled with NoSQL.

There's a licensing benefit too. My understanding of the GPL, influenced by my correspondence with the FSF, is that the small executables can be licensed GPL 2 or 3, and yet they can be used to build applications of any license, including proprietary.

I still have no form executable, and given the time required to write a good one, that might not happen for awhile. However, there are many ways to kludge a form executable, including a simple Vim interface. While such kludges might not be appropriate for my wife and daughter, they'll enable me to develop the rest of the app. When all is developed except the form executable, the value of the "Application Assembly using Small Executables" (AAuSE for short) concept will be proven, and a two week development effort on the form executable will be fully justifiable.

So kick back, relax, and learn about a Unix-centric rapid application development method. And remember, if you use GNU/Linux, this is your magazine.
Steve Litt is the author of Troubleshooting Techniques of the Successful Technologist.   Steve can be reached at his email address.

Understanding the Concept

By Steve Litt
It's called AAuSE, standing for Application Assembly using Small Executables, and it's just what it sounds like. Assemble an entire application, including user interface, using small executables. It's a Unix thang...

The creators of Unix were geniuses. Unix comes with hundreds of executables that "do one thing, and do it well". Just a tiny group of examples includes:

  • at
  • test
  • time
  • date
  • ls
  • du
  • df
  • find
  • basename
  • cat
  • echo
  • cut
  • sed
  • sort
  • wc
  • uniq
  • grep
  • head
  • tail
  • diff
  • less
  • lpr
  • dd
  • cp
  • ln
  • rm
  • touch
  • mktemp

The creators of Unix included several ways of putting these tiny executables together to form programs performing substantial work:
These executables and connection methods can be used to build some pretty substantial programs, as long as those programs have minimal user interfaces. But in order to fortify these tools to the capability to produce full applications with user interfaces, you need three additional executables:
The dialog program could have performed all three functions, but for reasons described in a later article, I felt it would be better to write my own and so wrote them.

The Picklist->Form Architecture

From the user perspective, I like apps with a menu at the top level. Menus call picklists, which call forms. A field on a form calls a validation routine, which might call a picklist, from which the contents of the field can be selected. While in that picklist, records can be added, changed or deleted. Here's a graphic representation of that user perspective:
Picklist->form architecture
Here's a grapical representation of the transitions between picklists and forms:
x

As you can see, when the user inputs data, he starts at a picklist (recordpicker) to select the data row on which he wants to work, be it add, change or delete. Assuming change, he's brought to a form for that data row. On that form he can change that row's columns. On the form, the Proj Leader column must be the employee number of an existing employee, so upon exiting that field, the contents are tested for a valid employee number, and if it's not valid, the employee picklist is brought up. The user can choose an employee from that list, but the user can also add, change or delete an employee. This implies that picklists have multiple user actions represented by different keystrokes. Some possible user actions are select, add, change, delete, and quit.

Such a drilling down from list to form to list to form can go on many levels. The benefit of this user interface is that it's fairly intuitive to the user, even if the current form or list completely covers what was under it. This makes it ideal for non GUI applications, but also excellent for GUI applications.

The picklist->form->picklist interface, when implemented right, eliminates the "back out and go around" hassle that some apps require when inputting a row necessary to put on a form. It also limits the temptation for the user to open (and forget to close) multiple windows and instances of forms and picklists.

Implementation Using Small Executables

So far we've discussed assembly of software using small executables, and the picklist->form->picklist user interface. Now it's time to put those two concepts together. At the top level, an application would be a shellscript to run any initialization code and set and export necessary environment variables, then run the menu system, and run finalization code once the user quits out of the menu system. The initialization code can be shellscript code inside the application shellscript, separate shellscripts, Ruby, Python, Perl or Java programs, or even C or C++ compiled binary executables.

The menu is a single executable whose job it is to implement a menu system. One could be made with a Ruby program repeatedly calling the dialog menu facility with the proper data. I used UMENU, available on Troubleshooters.Com for the application's menu. My reasons for choosing UMENU are detailed later in this magazine.

The following graphic shows the implementation of the picklist->form->picklist user interface using small executables:

X
In the preceding picture, the blue lines show the user perception of application flow. In reality, Function Shellscript 1 controls the calling of picklists, forms, validation routines and any other data processing. Since successive forms and picklist cover each other, the best way to represent them is with a stack. There's a small executable to stack data, with pushes and pops. Such an executable is very easy to write.

The other thing Function shellscript 1 does is read and write data to and from the database. The interface to the database is a small executable provided by the database vendor. MySQL provides the mysql executable, while Postgres provides the psql executable. If you're using NoSQL, your shellscript will interface with the many executables provided by NoSQL. You can read about these executables in the April 2007 Linux Productivity Magazine.

In the preceding diagram, Function shellscripts 2 and 3 are portrayed in minimal fashion. Maybe they're more Picklist->form->picklist type interfaces, maybe they're reports, maybe they're batch processes (month end or whatever). Each functional shellscript is called by a specific item in the menu structure. If you're using UMENU, some of the simple functional shellscripts can be placed right into the UMENU choice commands, eliminating the separate shellscript. This reduces the number of small shellscripts, and can provide a more straightforward system from the programmer point of view.

As mentioned in the Editors Desk article, I've already written the picklist executable (I call it recordpicker, and the filename of the executable is rpick). If you're not particular about aesthetics and you don't need the capability of picking multiple items from the picklist, rpick will serve you just fine. Otherwise, you can make your own, possibly from dialog.

The only major executable that remains unwritten is the form executable. Until someone writes this executable, it's probably best to kludge it with a shellscript that feeds into Vim and communicates via an intermediate file. Such an interface won't work for the nonprogrammer user, but it can help you get your application written, after which you'll have the justification to spend more time researching a better form solution.

About dialog

By Steve Litt
There's a program called dialog that can perform all three functions:
Use it if you want, but from my brief look at it, it didn't appear up to the job. First, remember the edict "do one thing and do it right"? dialog can make a form, a menu, a picklist, a passwordbox, or a gauge. Neither the man page nor the --help option adequately explained how to use it. It appears to use stdout to return information, requiring the application programmer to do various tricks with redirection. Beyond that, dialog is decorative and in so being, squanders screen real estate. My feeling is that the application programmer just might need every row and column on the text screen. So although I could have taken the time to study and master dialog, for all the reasons in this paragraph I decided to create my own recordpicker and filepicker.
Steve Litt is the author of Twenty Eight Tales of Troubleshooting.   Steve can be reached at his email address.

Why UMENU is So Productive

UMENU is extremely productive both for the user and for the programmer. Here's why:

For the User

The user operates UMENU with keystrokes alone. No mouse needed or recognized. No need to press Enter after pressing your key of choice. Choosing a choice on a menu 4 menus deep would be 4 keystrokes. For a touch typing user UMENU is THE fastest menu interface around.

UMENU isn't for every user. The nontypist will mourn the loss of his mouse. The user who has bought into Bill Gates' "users are stupid" propaganda will feel lost without his colors and icons. The user demanding aesthetics will be very disappointed with UMENU.

But the user who can type and values utility over fashion will use UMENU to run circles around his compatriots using other menuing systems.

For the Programmer

EMDL and VimOutliner!

EMDL stands for Easy Menu Definition Language. It's just what the name says, and because it's implemented as a tab indented outline, it can be edited with VimOutliner. VimOutliner is the fastest outline processor in the world, and anyone knowing the Vim editor can use VimOutliner with ease.

Using VimOutliner to edit EMDL, it would take a programmer about an hour to create a menu system with 25 commands. That's a small but practical app. In a 10 hour day the programmer could create a 200 command menu system -- enough to oversee a rather large application.

The real beauty of EMDL comes when you need to modify the existing menu system. Whole subtrees can be collapsed, copied, moved. Small subscripts can be incorporated right into a menu choice's command structure. A major rearrangment of a 200 command menu could take less than an hour.

Adding a new executable to an existing UMENU is a two minute task. The programmer can put in multiple lines of command information, and the EMDL parser will concatenate them (so be sure to use semicolons at the end of command headlines). The programmer can define environment variables and add directories to the front of the path for a given menu choice. Using UMENU's Prompted Argument Substitution feature, the menu can be programmed to ask the user for data.

Creating the menu hierarchy for a new application helps the programmer think about the needs of the app, the screens that must be created, and the data that must be captured and stored. It's an easy and important design activity.

During development the programmer can have a "Developer Menu" menu tree right in the application. UMENU EMDL editing and EMDL recompilation can be two of the choices, making menu updates a couple keystrokes. C program recompiles can also be put in the Developers Menu. Various tests can be incorporated in the Developers Menu. When it comes time to deploy the app to users, the programmer can delete the Developer Menu subtree, copy it to a different file, and recompile UMENU. If later debugging is needed, the Developer Menu can be copied back into the main menu.

What UMENU Looks Like

By Steve Litt
Envision this. You want a text app (non-GUI app) to play music. It has to work with playlists. As far as you know, none exists. You don't have a lot of spare time to create a computer program. What do you do?

Answer: You use UMENU as a front end to mplayer.

You've heard that mplayer has an API so that front ends can be built for it. mplayer is an interactive sound (and video) player capable of playing song playlists. If you know the keystrokes comprising mplayer's user interface, it's all you need. Good luck with that -- its interface is unintuitive to the point of infathomability. So you're going to build a text mode front end for it.

You read a little about mplayer's slave mode. The slave mode is where it takes commands, terminated by newlines, from stdin or from a FIFO. A fifo is a special kind of file whose purpose is to be written at the end from one program and read from the beginning from the other. It's created with the mkfifo Linux command. Your front end will write commands to the FIFO, and mplayer will read from it.

Here are a few example mplayer slave mode commands:
Command Purpose 
echo volume 8 > steve.fifo
 Play the song a little louder (repeat as necessary). 
echo volume -8 > steve.fifo
 Play the song a little softer (repeat as necessary). 
echo mute 1 > steve.fifo
 Mute mplayer (song keeps playing silently). 
echo mute 0 > steve.fifo
 Unmute mplayer.

For a more complete list of slave commands, click here.

You verify that these commands work fine. The only problem is, you sure don't want to type all that stuff every time you want to change the volume or move ahead to the next song.

Luckily you've installed UMENU. UMENU is a text mode (Command Line Interface, CLI for short) menu system capable of launching an entire command with a single keystroke. UMENU menu hierarchies are built with Easy Menu Definition Language (EMDL). EMDL is a protocol that's basically a tab indented outline with certain conventions (see UMENU and EMDL links in URLs section of this magazine).

So you pull up your master EMDL file in VimOutliner, create a menu hierarchy using the commands listed above, run the EMDL to UMENU converter, transfer the new UMENU files, and bang, five minutes later you have your mplayer front end. Here are some screenshots of your new menu system:

Song player main menu     Song player main menu: Pause and unpause, and access submenus.
Song player playlist submenu Song player playlist menu: So far only Change Playlist is implemented, but Edit Playlist could be bolted to many programs, including Vim.
Song player seek submenu Song player seek menu: Here's where you can move around the current song as well as moving to the previous or next song on the playlist. You can also jump to any song in the current playlist, and you can change to a different playlist.
Song player volume submenu Song player volume menu: Here's where you adjust the sound. L increases a couple DB, S lowers by the same amount. By quickly repeating keypresses, you can quickly adjust the volume as needed. Also includes mute and unmute.
Special Functions Submenu Special Functions menu: You can load mplayer from here, or unload (zap) it. You can also edit the music player's menu system and rebuild (compile and transfer) the system, and you can see the current time.

Also present is a submenu for performing FIFO tasks.
FIFO submenu FIFO Task menu: You can delete the FIFO or create a new one. These are seldom done. You can also kill orphan mplayers to fix situations where multiple mplayers are loaded. One could argue this should be on the parent menu (Special Functions Menu) instead. No problem, it would take two minutes to make that change.9

Life is good. In a matter of minutes you bolted a UMENU interface onto mplayer to achieve a small footprint song player application.

The Filepicker

With only UMENU and mplayer, you now have the text song player of your dreams. Well, except for changing playlists. UMENU asks you for the filename and you type it in. You'd prefer a filepicker that you can navigate across files and directories to choose the right one.

So you download the filepicker.
File Picker     Here's the filepicker, exploring the /etc directory. The current file is Muttrc. The files preceded by slashes are directories, and selecting them plunges you into the directory. Selecting the /.. directory escapes you to the parent directory (in the case of /etc, that would be the root directory).

The first version of this software reads only printable keystrokes and control keys, so it has a vi like interface: j goes down, k goes up, y selects, q quits.


The filepicker is a standalone executable receiving information (namely the starting directory) inside a file whose filename is its one and only argument, and passing back information (the name of the file selected, and the user's intent (select or cancel) in the same file from which its incoming information came.

So what you do is have UMENU create a temporary file with mktemp and store the temporary file's filename in environment variable $TMPFILE. Next UMENU writes the starting directory to that file. Then UMENU calls the filepicker with argument $TMPFILE. The user does his thing picking the file, and then UMENU reads the temporary file and assigns the name of the chosen file to $FN. Finally, if the temporary file reveals that the user selected rather than cancelled, it runs the command to switch to the playlist whose filename is in environment variable $FN.

The filepicker just bolts right on to UMENU with the thinnest possible interface -- a single file. You can bolt the filepicker onto any shellscript, or even a C, Perl, Python, or Ruby program using a system() call.

Have you gotten cynical over the years when those preaching "code reusability" write code that can't be reused, and is also overly complicated? UMENU and the Filepicker can be used over and over again as major components of various interactive applications.

The filepicker is written and running on my system. It will soon be released as free software -- probably GPLv2 license. It's a C program with no special libraries or dependencies. If you use Linux, this should quickly and easily compile and run on your system.

The Recordpicker

Now that you have the text song app of your dreams, wouldn't it be nice to be able to navigate the playlist by picking a song from the list rather than repeatedly choosing "Previous" or "Next" until you get the right song? You can, with the recordpicker and a little glue code.
Recordpicker     To the left is an example of the filepicker. The current "selection" is indicated by the equal sign and right angle bracket at its left. With the current keystroke definitions, you move down with j, up with k, select with y, quit with q.

Longer moves can be made with Shift+J (1 screenful), Ctrl+J (20 screenfuls), Shift+K and Ctrl+K, as well as g (top) and G (bottom).

Notice the records are not sorted. It's up to the calling script to order them as appropriate.

The recordpicker uses a lot of the same code as the filepicker. However, to make the recordpicker truly useful, the next version will have keystrokes defined by the calling script and passed in via the intermediate file. That way, you can navigate to a record and not only add, change or delete, but you can also print it, or anything else you want to do with it.

Using the recordpicker, you cat the current playlist into the temp file, allow the user to pick the song of his/her choice, and then increment the playlist position to that song's position. You might need to keep a persistent record of the filename of the current picklist, but that certainly isn't difficult.

The Formmaker

NOTE: This hasn't been written yet.

So there you are, with a universal menu front end (UMENU), a filepicker and a recordpicker. You can pretty much write a set of simple scripts and programs, and then turn them into an app with UMENU, the filepicker and the recordpicker. You need only one thing to make your application construction set complete -- a formmaker.

Once you have a separate executable acquiring field information from the user, you can combine it with the UMENU, the filepicker, the recordpicker, and a datastore to make any desired app.

If your data is stored in NoSQL (see the April 2007 Linux Productivity magazine), your task is easy indeed. NoSQL is made for this type of thing, and NoSQL has the exact same philosophy as UMENU, the filepicker, recordpicker and form executable: Let Linux do the work.

You could use mysql or Postgres as your datastore, using the mysql or psql executables to interface with the datastore.

Please, please, please would somebody write the formmaker so quickapps become a reality?

Quickapps

The search for the holy grail dates back before my entry into IT. Some called it "reusability". Some called it "RAD". Some called it "Template Driven Development". Its concepts were referred to as "software factory" and even "egoless programming". It seldom worked, and was usually slower to develop than apps meant to be written only once. The problem, of course, was that most of it was hideously unmodular. Interfaces were thick, massively parallel and surprising. Unit testing became difficult enough to require special testing software. Test points were few and difficult to access.

Meanwhile, for the last 35 years, the holy grail has been right in front of our face. It's called Unix.

Unix (and hence Linux and BSD) is an operating system whose philosophy is to get things done using strings of processes, each of which does one thing and does it right. Each process has a single input and a single output and no side effects. Each process communicates with the other via an incredibly thin (and observable) interface such as:
Using the processes as blocks and the interfaces as lines between the blocks, you can make a block diagram of the desired software system, and build it like you'd build with LEGO(R) bricks.

LEGO is a trademark of the LEGO Group.

You can build bigger units from smaller units, but always strive to make sure the composite unit has one input, one output, and no side effects. If you must have multiple inputs or outputs, document them carefully.

What a Quickapp Might Look Like

You need to create a quickapp for inventory and fulfillment of your product sales. To make things simple all your products are from the same category and color isn't an issue. Here are the major data tables necessary:
Based on the preceding, the application's menu structure (of course implemented in UMENU) becomes pretty obvious:
It takes about 45 minutes to create the preceding menu system with all the sorts, although of course the actual commands run from the menu would be dummies at this point.

Here's a reminder of the tools at our disposal, assuming the datastore is Postgres:
The reports are all simple calls to psql, or perhaps if formatting is important calls to formatting scripts which in turn call psql.

The single table data maintenance items each perform a psql query of the proper table and order, write it to the temp file, and call the recordpicker. From the recordpicker, the user chooses to add, change, delete or cancel. Add, change and delete all call the formmaker to do the work, and then information is passed back up the stack through the temporary file.

The multitable data maintenance apps are probably best implemented as Ruby, Python or Perl scripts spawning psql to interface with the database. Why not access the data directly from the scripting language via something like DBI::DBD? For simplicity. You're segregating database access from program logic, and that's always a good thing. Anyway, the Ruby, Python or Perl script calls the recordpicker as appropriate. For things like multitable edits, the "Edit routine" used by the recordpicker can be defined via info in the temp file to a special script, instead of letting it default to the formmaker. And of course, the formmaker has hooks for pre and post field validation scripts that can be defined by entries in the intermediate files.

The activities are Ruby, Python or Perl scripts that perform the required tasks, using psql and the filepicker, formpicker or formmaker as necessary.

Wrapping Database Access

For security reasons psql might not be adequate, especially when the database requires user authentication with password. You sure wouldn't want the password readable from a ps command. In that case, you could have a separate executable to interface between your quickapp and psql, or interface directly to the database. This app would have a persistent connection.

Living in the World of Quickapps

Your assignment is to create a six table app. Word on the street is there's not a lot of money for the project and it needs to be done quickly.

You take 4 hours to figure out the form of the database -- what info goes in what tables. Then you put all simple table maintenance into UMENU, with UMENU calling the recordpicker which itself calls the formmaker. Leave out security and validation for the time being. In the last hour of the day put the reports on the maintained simple tables into UMENU. Just before you leave, show the boss what you've done.

The boss might jump for joy, or she might say "OK, so far so good", but you've programmed a credible promise of a completed app, so she won't fire you or cancel the project. You've lived to program another day.

The next day do a couple of the complex data maintenance items (such as order maintenance, which depends on items, products and customers). Show it to the boss, and make the point that no, it isn't as user friendly or secure as it will be, but after two days a large portion of the application is running.

The next day finish the complex data maintenance items, and if there's time start on the activities. The next day or two finish the activities and tighten up security. Have the boss put a couple people on the new app to test and make suggestions.

From that point it's just like any other app in the maintenance part of the development cycle, except that because this app is made from small shellscripts and UMENU configurations, it's incredibly quick and easy to change. The only unchangeable things are the actual workings of the pickers and the formmaker, but this is primarily user interface choices. If need be, you can often configure the shellscript glue code to make it look more like what the customer wants.

Where Quickapps Don't Work

If the boss has her heart set on Java, or C#, or Windows, or GUI, a quickapp won't work. She'll need to pay full price for full scale development. If she insists on an aesthetically pleasing product, a quickapp won't work. Steve Jobs would not approve of this type of quickapp.

If the user community has their heart set on GUI and your boss goes along with it, quickapps won't work. If the users buy iPhones and iPods and Macs because of their grace and beauty, or buy their cars because of the image the car gives them, and the boss goes along with their wishes, quickapps are out.

The quickapp philosophy is like the original Volkswagen philosophy -- get from here to there as simply and economically as possible. Contrast that to today's car commercials emphasizing sound systems, magnesium paddle shifters, and being the hammer instead of being the nail sticking up.

Today, there's no shortage of people ready, willing and able to go into debt for features irrelevent to the system's core functionality. If such people make development decisions, quickapps don't cost enough.

Where Quickapps DO Work

Many small business owners are just like me -- cheap! They won't commit to a three month development cycle before seeing the result. They WILL commit to a development cycle in which significant functionality is finished the first day.

Past development methods made it impractical to program for small businesses, except for one size fits all "vertical apps". The quickapp makes it economically feasible to program for a small business.

Another use of the quickapp is the "tideover" app. This is where you spend a week making a simplistic application, so customers are satisifed during the several months it takes the "real developers" to create their introspective, reflective EJB monolith that's as much a monument to programming as a tool for the users. I've done at least two tideover apps and got paid well to do them.

Yet another use is the "off the books" app. Either you or another employee detects a need, but there's no time or budget to address that need. So you spend 8 hours writing the quickapp, telling the boss you were working on other projects. I've probably done about 50 off the books apps. One three day app was celebrated as saving several employee-hours and a box of computer paper per day. A week's worth of programming to fix a scanning problem morphed into the timesheet front end used by a major law firm for the next five years.

Of course, sometimes politics come in -- a two day app, finished and delivered, to give accounting a specific piece of information without going through five different processes was vetoed and retracted because it wasn't preapproved, documented, and studied for deployment problems. I got in some trouble for deploying it.

Perhaps the best use of a quickapp is for yourself. When you need software, you don't want to pay someone to write it, and you're not paid to write it, you need it done fast. Quickapp to the rescue.

When doing a quickapp, I'd suggest starting by calling it a "prototype" or a "workaround", and take your bows only when it becomes hugely popular. Also, be VERY careful not to do anything that can jeopardize security. Nothing should run as root. All intermediate files and fifos should be chmod 600. All user input should be checked to prevent SQL injection, buffer overflow and the like.

Quickapps Can Be Made Corporationally Correct

Quickapps are comprised of a menu handler, a filepicker, a recordpicker, a formmaker, and a database handler. Those elements are glued together with files, FIFOs, pipes, backticks and sockets. There's nothing preventing you from developing versions of these elements that derive input from mouse clicks and output html. There's nothing to prevent you from developing Perl-Tk versions of these elements yielding a nice, pretty, mouse aware quickapp. These elements could be developed in Java as well. Once such elements are created and tested, such quickapps would still develop lightning fast.

In the HTML version, you'd need some extra elements to guarantee security, and also an extra element for persistance (HTTP is stateless), but that's not a big deal.

Don't Judge This Book By Its Cover

UMENU is a text only program. It moved the cursor only forward, never backward. Screen erases are accomplished by 25 newlines. It's monochrome. It's ugly.

My filepicker and recordpicker are text only programs. They move the cursor only forward, never backward. They erase the screen via a bash reset command. Every time you move the cursor up or down, the entire screen is rewritten. It works, it's fast enough, and is perfectly legible and useable, but depending on the hardware and/or terminal emulator, there's some screen flicker. They're monochrome. They're ugly.

Please don't make the mistake of judging the paradigm called "Application Assembly using Small Executables" (AAuSE) by the aesthetics of my implementation of the paradigm. My implementation is governed by my priorities, which might not be your priorities. My priorities are "easy to implement, small footprint, runs anywhere, uses screen real estate efficiently". Aesthetics isn't even on my radar. The ugliness of my implementation says nothing about the paradigm.

In fact, it would be easy to incorporate nCurses into UMENU, my Filepicker and my Recordpicker.  The only reason I didn't incorporate nCurses into the two pickers was because the default nCurses compile makes it difficult to test for memory leak.

If you want to go prettier than nCurses, you could probably implement a menu and both pickers using dialog. You'd need to write some interesting glue code and you'd sacrifice some screen real estate, but I'm pretty sure it could be done.

Not pretty enough? Do them in Perl-Tk. You give up the ability to work in a CLI environment, and window management might present a few challenges, but you can implement Application Assembly using Small Executables as beautifully as you desire.

Perhaps you want to make webapps. Your executables could each write an HTML page. State would present challenges, but of course this is true of any webapp. You could probably write a cookiemanager executable to handle state.

The point is, don't judge the programming paradigm called "Application Assembly with Small Executabes" by Steve Litt's implementation of the paradigm's executables. I'm well known for favoring "Army Surplus" apps -- simple monochrome, run-anywhere apps that just work. The paradigm can be implemented with YOUR priorities if you spend a little time. And the beauty is a lot of my code -- the non-presentation part -- can be used to implement your priorities.

Refuting the Rotten Tomatoes

Using AAuSE (Application Assembly using Small Executables) will not make you popular. People will hate it, and not just the stridently corporationally correct. I recently showed my music player to one of the geekiest guys I know, and to say the least he wasn't impressed:
Allow me to answer those objections...

It's Ugly!

As detailed in the Don't Judge This Book By Its Cover article in this magazine, ugliness isn't a property of the AAuSE paradigm. It's a property of my implementation. If your priorities are different, start with my implementation and make the presentation prettier.

It's Not Real Programming!

So? It gets the job done fast. Real programming or not, the result is useful, and often less bug-ridden than the application written from scratch or assembled from other people's libraries.

This argument reminds me of the Mainframers' anti-Dbase arguments of the early 1980's. Yeah, sure you can whip out an app in a day using Dbase, but it's not real programming. I remember those Mainframers all too well. The Mainframe Manager asked me to write a program to convert her Cobol output for use on the PC local area network. There was one piece of necessary information the mainframe wasn't passing, so I asked her to pass it to me, and she balked. I said "you CAN pass it to me, can't you?"

She said "Welllll, we could, but to do so we'd have to write a program!"

This lady had at least three programmers working for her, and she was worried about the resources needed to write a program to supply a piece of information with the Mainframers' "real programming" techniques, while because of my "quick and dirty" Turbo Pascal and Clarion programming tools, writing a program could be as little as a two hour project. When I hear "it isn't real programming", I can't help but remember that situation.

Some things never change. The big-iron guys, whether it's a three million line Cobol program on an IBM370 or an Enterprise Javabean Java Application on a muscular Linux box, the big-iron guys don't like techniques of the nimble small-iron guys who can write a program in a day.

Programming can be a hobby. It can be an excellent hobby. But it's a hobby nobody's going to pay you to do. When you get paid, it's to produce a high quality solution to a problem. Quickly! If that high quality solution isn't a "real application" produced by "real programming", it might not matter to the person signing the paycheck. Even if you're programming for your own business, such programming isn't billable hours. If you can quickly produce an automated solution to an internal problem and then get back to getting paid, you're not going to relinquish that opportunity because it's not "real programming" or because it's not a "real application".

Besides all that, AAuSE doesn't preclude "real programming". At the center of every application is heavy duty logic to operate on the data in a unique way. You use "real programming", in Perl, Python, Ruby, Java, C#, C, C++ or even assembler, to write that core of heavy duty logic. You write the core as a standalone executable without the encumberance of a user interface. Then you bolt on menus, picklists and forms as separate executables to supply the core logic with its data and configuration.

Sooner or Later You Need to Write Code!

We all know that. See the last paragraph of the preceding section.

It's Soooo 1975!

So? It works!

Not everything old is bad. Otherwise there would be no political conservatives. This magazine is called Linux Productivity Magazine, with Linux being a workalike to the Unix operating system first developed in 1969. Apparently you like Unix more than the much more modern Microsoft Windows. What AAuSE does is use the small executable philosophy built into Unix to ease application construction.

And let's examine why simple, keyboard driven interfaces fell from favor. They certainly didn't fall from favor for productivity reasons. Everyone knows that a touch typist can run circles around a mouser.

Perhaps they fell from favor in order to accommodate the stupidity of users. We all know how stupid users are. But I'm old enough to remember the days of PDP-11/RT/11 and DOS, and I can tell you, after a few hours training an average person could work a keyboard driven app just fine. They didn't need icons, or colors, or mice. Within a few days their superior productivity paid back that couple hours of training, and for the rest of their employment, the rest was gravy.

Tell me one more time just so I understand: Why does a spreadsheet need to be GUI? What was so bad about Lotus 123? Why was Excel necessary?

The simple answer is marketing. The state of DOS in the 80's was such that anyone could write an excellent keyboard driven app for it. Microsoft had no advantage. Indeed, Microsoft was at a disadvantage because other companies did better programming. A trip back to 1988 would show the leading spreadsheet to be text mode Lotus 123 from Lotus Development, and the leading wordprocessor to be text mode WordPerfect from WordPerfect Corporation. If Microsoft wanted to extend its leadership beyond the operating system itself, they had to redefine the contest.

So Microsoft introduced Windows, and simultaneously marketed the concept of people being too stupid for a keyboarding interface. It worked and the world switched to Windows, which was much harder to program for. Whereas in DOS the user interface was maybe 10% of the programming, in Windows it was as much as 80%. And worse, the US Justice Department accused Microsoft of failing to disclose the most efficient parts of the Windows API, so that using those efficient parts Microsoft could always create better software than its competitors.

To reiterate, the main reason we use mice and GUI today is marketing, not utility. Software had a speedier user interface in 1975.

You Can Do the Same Thing With Libraries!

Oh this is so seductive. The argument goes something like this:

Hey Steve -- you can do the same thing with good libraries in a good programming language once you learn the libraries. You have a library for picklists, one for menus, and one for forms. Or maybe one that does all three. You write your software and wrap those libraries around it.

Sounds good, doesn't it. You can abandon the kludginess, the corporational incorrectitude of separate executables just by putting that separation at the library level instead of the executable level. The best of both worlds!

Trouble is, the preceding argument overlooks the many advantages of individual executables. Let's start with memory management. Memory leak cannot extend beyond the scope of the executable. Check out grabmem.c on the left below, and grabmem.sh on the right below. Grabmem.c mallocs and does not free a million bytes. Grabmem.sh calls grabmem.c (as its compiled version grabmem.bin) 10,000 times, so 10GB of RAM is allocated but never freed. Run grabmem.sh on one terminal, and vmstat -n 1 10000 on another. You'll see that system memory changes little because grabmem.c gives back all its memory on executable termination.

grabmem.c     grabmem.sh 
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <memory.h>
void grabmem(){
	long bytes2grab=1000000;
	char *buf;
	buf=malloc(bytes2grab);
	assert(buf != NULL);
	memset(buf, 's', bytes2grab);
	printf("%ld more bytes grabbed\n", bytes2grab);
}

int main(int argc, char *argv[]){
	grabmem();
	return 0;
}
    
#!/bin/bash
let repeats=10000
let iteration=1
while test $iteration -le $repeats; do
	echo -n "$iteration "
	./grabmem.bin
	let iteration+=1
done

As mentioned, the preceding shellscript/executable combination reduces free memory a tiny bit, and then it stays constant throughout the shellscript. On shellscript termination it goes back up.

Now make it monolithic by putting the loop inside the C program and running just the C program:
##include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <memory.h>
void grabmem(){
	long bytes2grab=1000000;
	char *buf;
	buf=malloc(bytes2grab);
	assert(buf != NULL);
	memset(buf, 's', bytes2grab);
	printf("%ld more bytes grabbed\n", bytes2grab);
}

int main(int argc, char *argv[]){
	long iteration=1;
	long repeats=10000;
	for(iteration=1; iteration <= repeats; iteration++){
		printf("%ld ", iteration);
		grabmem();
	}
	return 0;
}

What a difference. This time free memory keeps going down until it reaches a very low figure and the program slows to a crawl, as all subsequent memory operations happen in swap memory. On my machine the program finally aborts when the assert() after the malloc() fails. For several seconds after the program termination, the entire computer is pig slow as swapped memory is slowly restored to free memory.

The library advocate might say you simply need to be careful not to leak memory. Fine, if you're that careful, skillful and exquisite a programmer. But are you going to have to go back into all your third party libraries and debug leaks in them too?

Individual executables provide a fire lane across which memory leak cannot jump. You might still have leaks, but probably not the kind that terminate the program.

Now let's talk about complexity. If you've written code for more than six months, you know that code complexity, and the difficulty of writing it, is not proportional to the number of lines of code. It goes up geometrically with lines of code because complexity is proportional to the interactions between all the variables and subroutines. Thus a 1000 line program is probably a couple days of programming. A 10,000 line program is couple months. A 100,000 line program is several programmer-years, is not doable by a single person in a reasonable timeframe, and stands a substantial chance of never being completed.

An application built of separate executables is much more proportional because of the "buck stops here" nature of an executable. By its very nature an executable is extremely modular. It doesn't mess with other executables' memory, variables, objects or subroutines. It writes only files it's intended to write. The opportunity for interactions and entanglement is low. It has only a very small number of ways to pass data:
With the exception of sockets, the preceding methods are trivial. An executable can always be tested in the absense of other application components. Its input or output can be intercepted or changed. It's so small and simple that bugs cannot find a place to hide. It's easy to create a workalike executable that "plugs in" in place of the original.

Compare this to libraries. Each library has a fairly complex API. Interactions are often groups of pointers to pointers to pointers. It's easy to become confused.

It's Slow!

So true. Code written in C is orders of magnitude faster than code written in bash. This is especially true in loops, where bash must run a new program in each iteration. You would never use bash to create a prime number generator.

But bash is more than up to the task of creating a user interface. Modern computers are fast enough that even the fastest typists can't get ahead of bash code.

And then there's the argument that "You can't do threads with bash!". Well yeah, you can't. So what? Tell me again why I'd want threads throughout my user interface code? Remember, the picklist->form->picklist pattern alleviates the need for multiple sessions of the app. Anything you need to fix from a form filed can be fixed by drilling down from that field to its picklist to a form. It's every bit as quick as running another session and navigating to the to the relevent picklist, and it's a heck of a lot more intuitive.

But what if you want the program to continue processing while you're operating the user interface? I'm not quite sure how much time that saves. If user interface consumes a minute and then the process takes an hour, the extra minute is immaterial. If the user interface consumes a minute and then the process takes a second, the process time is immaterial. The worst case would be if the process occurred in several long steps, each requiring user input. That problem was solved years ago by people creating Linux installers. Collect all the data first, put it in a file, and then let the installer work off that file, in the background, while the user goes on to other things.

But perhaps on some unusual app you must MUST have the process and the user interface operate concurrently. No problem -- the process can run in the background, with process and interface communicating through fifos, or sockets, or intermediate files with polling, or if you don't like polling then with Unix signals. This might not be as easy as Java multithreading, but it's rarely necessary at all, and it's doable.

And remember, bash is the glue language. The executables are written in a more appropriate language. If a part of an application must be incredibly fast (prime number generator, for instance), by all means write that executable in C. The Application Assembly using Small Executables paradigm provides the world's easiest way to mix computer languages.

So with AAuSE, outer loops may be slow, but their bottlenecks are the user. Inner loops are written in faster languages so they're fast. AAuSE usually produces applications that are easily fast enough.

You Cannot Pass Complex Data Structures!

Sure you could, if you wanted to. You can write intermediate data to a database. You can write it to a file. If the data structure is a hierarchy you could write a tab indented outline or an XML doc. You can write Ruby yaml. A linked list can be passed as a simple file of lines. So can a stack.

I can hear the agonized cries now: "But file I/O is soooo slow!" Refer to the last section of this article -- AAuSE links together a small number of executables to do the job. Usually transfers between executables wait for user input. Under those circumstances, waiting for file I/O is easily acceptable.

Remember my first sentence in this section: "Sure you could, if you wanted to."

Do you really want to pass complex data between executables? Probably not. The power of Application Assembly using Small Executables is the executables' inherent modularity, and the incredibly thin interface between them. Why gum that up with complex data structures?

Take the example of the recordpicker. Into it you pass a file containing the allowed user actions (with suitable defaults), and the records from which to select. Coming out of the recordpicker, the file contains the user's action (SELECT, CANCEL, etc), the zero based number of the record selected, and the string representation of the record selected. The calling program can then either parse the string to retrieve the record, or it could have saved the list of records and accessed it via record number, or it could rerun the SQL query, including the record number. No need to pass complex data.

If one uses AAuSE correctly, there's little reason to pass complex data between executables, and for those rare occasions when you need to, you can.

It Only Works For Simple Apps!

Define simple.

You can use this method to create an incredibly powerful statistics package. Let's say you need to run ten different types of statistical analyses on a data warehouse. Here's what you do. You hire a statistical programmer to write batch processes for each of the ten analyses. I'd prefer each process would be its own executable, but if that's not practical they could be rolled into one or a few.

The statistical programmer can keep his mind on the statistical problem domain, because his total interface is command line arguments and perhaps very simple intermediate files, or perhaps one executable piped into the next. All the statistical programmer needs to do is write simple documentation explaining what is expected on the command line and in any input or config files, and what can be expected to come out.

Then you hire me to create an AAuSE front end to the statistical programmer's work of genius. Using menu executables, recordpicker executables, form executables, shellscripts, and maybe a little Ruby and/or C, I create a front end to produce the input needed by the statistic programs, and to interpret and handle the output from those programs. I probably make the front end in a couple days to a few days.

Look what's happened. You didn't pay the statistical programmer megabux per hour to fiddle around with user interface code. You didn't restrict the statistical programmer's language, so he could use his most productive language. You reduced the complexity of the statistical programmer's task, thereby reducing the complexity of his code and giving bugs less nooks and crannys in which to hide. You've completely split the statistics algorithms from the user interface. Testing the statistical code is as simple as calling it, from the command line, with the right command line arguments and the right intermediate and config files. This makes testing much easier.

When someone says it works only for simple apps, I think what they really means is it works only for apps requiring only keyboard input, or in the case of Perl-Tk constructed menus, picklists, and forms, keyboard and mouse clicks. In other words, you can't use AAuSE apps to draw or drag. Oh, and you wouldn't write a wordprocessor using AAuSE.

Well Excuuuuuuuuse me!

Every technique has its limitations. The backtracking algorithm so good for running mazes and best-filling backup media is useless for ordinary database programming. The simulation techniques matching objects to cannonballs bog down horribly with business apps. The assembly language you use to write parts of drivers is useless in an accounts receivable package. And yes, if you absolutely need to draw or drag throughout the entire app (and not just in a couple parts of it), you shouldn't go AAuSE.

Otherwise, AAuSE just might be perfect for your needs.

Summary

Suggest Application Assembly using Small Executables, and the criticisms come out of the woodwork:
Yep, it's ugly, and if you're willing to trade lots of time or bucks for pretty, do that. Perhaps if you're really willing to trade that time, prettier menu, picklist and form executables could be coded so AAuSE produces a better looking app.

True, it's not "real programming" at the top level, but it's "real programming" in the areas that need "real programming", which is why "sooner or later you need to write code is true but not an anti AAuSE argument.

The argument about it producing being a circa-1975 user interface is true, and once again, if you're willing to pay top dollar for the 21st Century user interface created by Bill Gates' propaganda machine, by all means do so.

The argument that you can do the same thing with libraries is false. Yeah, you can assemble from libraries, but you lose a lot of the modularity that makes AAuSE so quick and robust.

A properly designed AAuSE interface is faster than a touch typist, so it's easily fast enough. I mean, would you want to do data input on a supercomputer?

You can easily pass complex data structures around, assuming you serialize them to files. Using XML and Ruby yaml techniques would even make it fairly easy, and of course many complex structures can be passed within the database used by the application. But this is sort of a moot point, because in the well designed AAuSE application, executables communicate with very thin interfaces, not complex data structures. If a section of the app needs to pass around complex data structures, that section should probably be a monolith, while the user interface continues to be AAuSE.

When someone argues that AAuSE works only for simple apps, they're defining "simple" very narrowly. What they're really saying is that the app as a whole requires input beyond what a keyboard can do.

You'll hear all these arguments. Be informed and make up your own mind.

Where to Use AAuSE

By Steve Litt
As a practical matter, you're probably not going to use AAuSE in work for hire as a contractor or employee. Management won't like its aesthetics. Besides, management probably wants a web app, and I can't even call them wrong. As difficult as statelessness makes webapp authoring, and as substantial their security risks, webapps are intuitive and can be used enterprise wide at any workstation having a browser.

Meanwhile, if at work the person you report to is or has been a programmer, he or she will probably hate the concept. They didn't spend years learning J2EE just to support simple executables and shellscripts.

The place to use AAuSE is on apps you will be using yourself. If you have a business, use it for your timekeeping, your inventory, invoice creation, tickler file and the like. If you have store bought programs for those functions, use AAuSE to add functionality to the store bought programs.

A great place to use it is for maintenance on your website database. If you're physically on the server, or if you can ssh to the server, an AAuSE app is likely easier than writing the equivalent functionality as a webapp, and it will certainly be a faster data entry environment.

There are situations where you can use it working for others. If you find yourself repeatedly performing the same SQL queries from the command line, why not take a few minutes to incorporate them in a UMENU structure to save keystrokes. Perhaps you can write the query results to an intermediate file, run a picker on them, and even edit one in a form executable. Maybe, just maybe, others will see the ease and simplicity of such "apps" and start using them, at least until the powers that be find out and shut it down for being corporationally incorrect.

Perhaps the best work for hire use of AAuSE is as a prototype. Whip out the app in one day, so that the Enterprise Javabean crowd can have a full six months to write it in Java. Your prototype can be used to enter test data, and it will be appreciated.

So use it for yourself, and perhaps at work for one-off apps only you use, and for quickie prototypes.
Steve Litt is the author of Twenty Eight Tales of Troubleshooting.   Steve can be reached at his email address.

AAuSE or Rails?

To quickly create an app, should you use AAuSE or Rails?

The short answer -- I don't know because I have little experience with either. Anything I say here is speculation, which is what computer columnists do. So here we go...

Both AAuSE and Rails are designed to quickly code an application, and both do a good job of it. Rails codes a web application, meaning it's better looking, and available to anyone with a browser. If your application is for the population at large, AAuSE isn't appropriate, and Rails does a very nice job.

If the app is for you or for a few employees, AAuSE produces a faster, keyboard-centric user interface. It's probably easier to keep AAuSE secure, because access is through ssh rather than http.

If you want to do it halfway well, Rails requires quite a bit of learning. A "hello world" Rails app is fairly straightforward, but the plethora of Rails tools make for a broad learning curve.

On the other hand, assuming you know how to code a shellscript, you can probably jump into AAuSE instantly by using your existing Perl, Python, Ruby or even C skills to create executables. Whereas Rails implements an MVC (Model View Controller) paradigm to separate presentation, logic and business rules, AAuSE separates out presentation with simple, prewritten executables, while business rules are incorporated in field validation shellscripts. Speaking of field validation, my understanding is it's much easier to do in AAuSE than in Rails. Most Rails apps I've seen validate the whole form when the user clicks OK, although it's certainly possible to use AJAX methods with Ruby to validate at the field level.

AAuSE is likely to run on whatever Linux/Unix/BSD computer you're using, especially if the computer has bash. With Rails, you'll need Ruby, Rails, some sort of Rails-friendly webserver (or else tweak Apache into serving Rails apps). Rails apps are harder to deploy on the server, but of course once deployed on the server, they can be accessed by anyone with a browser. By the same token, AAuSE can be deployed easily on the server and accessed through ssh, always assuming permissions for executables and data are set for a group into which the user falls.

If you use Rails you'll be using one of the most corporationally correct technologies on the planet, and jobs will be available. If you use AAuSE you'll be using one of the most corporationally INcorrect technologies on the planet. It's doubtful AAuSE will ever get you a job.

So how do you choose? I think it boils down to three factors:
  1. Desired user interface?
  2. Existing familiarity?
  3. New construction, or legacy wrapper?
If you want a web interface, Rails is your choice. If you must run on Windows without cygwin, you must use Rails. If you want a fully keyboard aware interface, AAuSE is your choice.

Despite the chatter of Rails advocates, Rails isn't trivial to learn. Different logic goes in different places, and those places are not always obvious nor traceable. AAuSE apps are more intuitive. Also, Rails achieves its phenominal productivity through default data field naming, so creating the database can be a little more difficult, and of course if the database is preexisting, you'll need to override Rails' defaults.

AAuSE is excellent at wrapping a user interface around an existing, and possibly quirky program. I think you can do that in Rails, but it's not obvious the way it is with AAuSE.

My opinion -- both will come in handy, so you should know both.

Menus and Me

By Steve Litt
Menus and I go back a long time. Back to the spring of 1989, specifically. My large law firm client asked me to write a quick and dirty menu program to tide them over until their head programmer finished the menu program they really wanted.

Two weeks later my menu program was done. Up to 24 items arranged in two columns, white on a blue background as I remember. Each item could be selected using arrow keys and the Enter key, or simply by typing the first upper case letter of the item's description -- no need to press Enter. The menu system was configured by editing a hierarchy of .mnu files. Each .mnu file was a simple text file edited with any editor. The menu program was lightning fast for a touch typist. After using it a couple weeks the menu program became part of my life and then part of my soul. I used it at work and at home.

A few weeks later my program was decommissioned when the employee delivered his program. His program allowed specification of directories to add to the front of the path, and the ability to specify environment variables. I was jealous. But his program required the user to press Enter after pressing the hotkey for the item. I protested that slowed me down. He said "big deal". To me it was a big deal, and I vowed never to make a user press Enter just to register a single keystroke. I also made it a goal to include pre-pathing and environment variable specification in any future menu program I wrote, as well as a killer feature of the old WordPerfect menu system: prompted argument substitution.

NOTE

Prompted argument substitution is a feature whereby a menu choice can be configured to ask for an argument. For instance, a time tracking program might ask for the project name. After you type it in, timing begins on that project.


Time passed; my wife and I were the only people on the planet using my menu program. When I gave her a new computer, her first question was "where's my menu?". I had to put the menu system on right away.

Days of DR DOS

I got the DR DOS operating system in the early 1990's. It was pretty much a complete MS DOS clone, plus it was multitasking. My menu program couldn't do multitasking because different sessions' temporary files clobbered each other. I modified the program to create temp files with unique names, fixing the problem. My menu program was now an everyday part of my business.

My Second Menu Program

In 1994 a courtroom software company contracted me to do software documentation. It became obvious they needed a menu system with which to run all the programs they offered their customers, and their one and only programmer was much too busy to do it. I did it.

This new menu program was much more special purpose than my first. It was tightly integrated with their software tools. But once again, the user didn't need to press Enter after pressing the letter identifying their choice. Writing a menu program the second time was easier, the product was better, my client was happy, and I realized I could write a menu program pretty much at will.

Hello Windows, Good Bye Menu Program

As a touch typist, I never liked Windows (or Mac). Reaching for a mouse is such a productivity killer, and it's so unnecessary in most contexts. But as Windows took over the world, it became a necessity. By 1994 Windows was on my main computer.

Windows has its own menu system, accessible by clicking the start button. A mouse driven menu system isn't nearly as quick, but with Windows it's impractical to call my menu program. What would I do, grab the mouse to run the keyboard friendly menu? Or press a series of arcane keystrokes? My menu program became yet another casualty of Windows.

What do you do when a beloved computer feature is taken away? At first you scream in protest as you fumble through the less efficient interface. You swear you'll go back to the old way (DOS), but realize that puts you out of sync with the whole world. So to justify your decision not to go back, you enthuse about the new way's new features. "Oh, these truetype fonts are wonderful!" "This object linking and embedding is spectacular!

Finally the day comes where you're used to doing it the slow way. Logically you remember there was once a faster way, but you don't feel it in your gut. You accept the theft of your feature.

This scenario played out all over the computer world as touch typists switched to the crashy, mouse-requiring world of Windows. They accepted, but in a remote corner of their brain, they remembered the better way.

Linux Boasts

In the April, 1998 Troubleshooting Professional Magazine, I made the following boast: "Don't like Unix commands and shellscripts? Make a menu to run your machine. Heck, if I get 50 people wanting it, I'll make the menu program myself.". Quite a boast for a guy who had never used Linux in his life.

I started dabbling with Linux in October 1998. November 1998 found me joining a LUG and becoming a recognized and popular Linux advocate. Having few Unix or Linux credentials, my LUG conversations increasingly centered around my programming prowess and what I was going to code now in Linux.

Four months later, in March 1999, my trash-talking had reached an artform, and it was time to put up or shut up. I took two days out of my busy life and coded a GPL licensed menu system in Perl. It was called it UMENU, and impressed several people at the next LUG meeting.

UMENU was single-letter press like my previous, work-for-hire menus. Like the menu system of the head programmer at the law firm, it enabled pre-pathing and also had hooks for later insertion of environment variables. Like the ancient WordPerfect menu system, it had prompted argument substitution.

And it was ugly! Both my work-for-hire menus incorporated libraries I'd writtten, as work-for-hire, to give nice window borders, move the cursor all over the screen, and enable color. Obviously it would have been a copyright violation to include those libraries, and the two days devoted to UMENU creation didn't allow writing new versions of those libraries. UMENU was (and still is today) a monochrome app writing printable characters to the screen via stdout, and acquiring keystrokes via stdin (suitably modified so as not to require the Enter key). It looked like something you'd buy at the Army Surplus Store.

Many saw this ugliness as a downfall, but the coin had two sides. UMENU's simplicity made it runnable just about anywhere with Perl 5. People started using it. Not a lot, but some. The feedback was good.

Suitably outfitted with a GPL version 2 license and a website of its own, UMENU became my first Free Software offering.

UMENU; Yawn

My fellow LUGsters now knew I could put my code where my mouth was. UMENU had done it's job: Given me Linux cred. But desktop use of Linux required a GUI environment, thereby making keystroke menu useage difficult. I used UMENU for a few special tasks such as easing kernel compilation and a chess appliance for a school, but in reality it was no easier to use than Red Hat's start button menu. 2000-2001 at Troubleshooters.Com, UMENU was basically shelfware.

The Practical UMENU System Menu

After switching full time to Linux in March 2001, by February 2002 it was obvious the silly graphical start menu was slowing me down. I wrote the following email to my LUG:

I'm getting tired of graphical menus and would like to use UMENU as my menu system. Problem is, if I have to mouse to it that defeats part of the purpose of UMENU. Does anyone know a way I can assign a keystroke which will always return me to the one and only UMENU session running?

SteveT

A few days later I discovered how to tweak IceWM to run an arbitary command when pressing Ctrl+9, and modified UMENU so it had an option to terminate upon running a command, and I had a practical CLI menu system for a GUI desktop.

The remaining problem was the difficulty of reorganizing the menu system by editing individual .mnu files. Small .mnu files are necessary for quick instantiation and small memory footprint, but authoring and reorganizing would have been much easier as an outline, so I defined EMDL (Easy Menu Definition Language), a protocol arranged as an outline which could be quickly written and modified via the VimOutliner outline processor. A couple days Perl programming created the EMDL to .mnu converter.  The converter made designing a menu system as simple as outlining a term paper.

With this converter, it was now easy to put most the programs and commands into the menu system. Because EMDL allows multiple command lines, you can code small scripts right in the EMDL file.

One of the first commands I put into the menu system was the EMDL editor (VimOutliner) and converter. The converter is a Perl script with tons of command line options. Front-ending it with VimOutliner made it memorable, easy to use, and intuitive.

So that was it. The Ctrl+9 key combination brings up UMENU, and from there the whole menu hierarchy can be navigated until the desired command is run, after which that instance of UMENU will terminate.

 After 10 years of graphical "Start Menu" work, at first it seemed wierd to once again use a keystroke menu system. But the more time passed, the more I liked my UMENU system to the exclusion of the Start Menu.

The Universal Front End

My home-office work on a an April 2004 contract required a task time tracking system, so I quickly coded a program capable of creating, starting, stopping and summarizing time on projects. The exact functionality depended on arguments to the program.

I could have created a front end to the program, but have you ever noticed that more than half the work of making a program is the front end? Life's too short. Using this three choice UMENU submenu yielded a fully front-ended timekeeping app:

     E: End current task
...  S: Start new task
     V: View/Modify task log

The "start new task" choice yields yet another submenu (that's why the ellipses) with a listing of all the current tasks I'm doing. If there's a task called "bike repair" under a category called "personal", here are the 5 keystrokes required to begin it:

Ctrl+9:  Bring up UMENU
T:           Timekeeping menu
S:           Start new task
P:           Personal
B:           Bike repair

Folks, that's five keystrokes; one complex but fast, four simple. That's less keystrokes than required for most command line commands. It can certainly be done faster than the time required to grab a mouse and then restore home position on the keyboard. If you forget the command it's no big deal -- the menu system walks you through the drilldown.

And of course, frontending that simple command took less than a half hour, and if reorganization of the user interface was needed, it was a 2 minute edit of an EMDL file, a 10 second conversion, and a couple minutes to test the new user interface. Better yet, it was a user interface that was already intimately familiar, because I used UMENU every day.

Universal Front End, Part 2

The upgrade from Mandriva 2006 to Mandriva 2007 was very nice except one thing -- no Xmms. All  the music players available on Mandriva 2007 revealed they were all lacking in one way or another. Some couldn't edit playlists, some had HUGE memory footprints, some had confusing interfaces. What would have really been nice was a text interface music player that handled playlists. If there were suitable front ends to play or mplayer, they were hard to find.

A little reading about mplayers slave mode indicated the ability to send commands straight into a running mplayer session using a FIFO "file". So to start up mplayer in this mode you'd do this:
nohup mplayer -ao alsa -loop 0 -input file=steve.fifo -playlist $FN &
In the preceding, $FN holds the filename of the playlist to be played. If you want to subsequently raise the volume, you'd issue this command:
echo volume 8 > steve.fifo
To move to the next song in the playllist you'd do this command:
echo pt_step 1 > steve.fifo
So I put these commands and several others into a menu structure, incorporated that menu structure as a submenu in my overall UMENU structure, and shazam, I had a surprisingly useful and quick text mode front end for playing my musical playlists. The menu structure looked like this:

        Control Mplayer
            Load Mplayer
            Zap Mplayer
            pAuse
            Unpause
            Playlist ::: Playlist menu
                Change Playlist
                Edit Playlist
                ^Quit
            Seek through song and playlist ::: Song/playlist Seek Menu
                Ten seconds forward
                ten Seconds backward
                One minute forward
                one Minute backward
                Beginning of song
                Next
                Previous
                ^Quit
            Volume ::: Mplayer volume
                Louder
                Softer
                Mute
                Unmute
                ^Quit
            ^Exit

So once you get into the volume submenu, you can raise and lower the volume with repeated keystrokes. In the seek submenu you can get exactly where you want in the song, or go back or forward a song. This menu structure might not be optimal. That's OK, it's a 5 minute job to totally rearrange it in the EMDL file and then reconvert.

There was just one problem. On changing playlists, it was necessary to type the playlist's filename into UMENU's prompted argument substitution prompt. It would have been nice to have UMENU acquire the playlist name via a filepicker...

The Filepicker

After tiring of typing in filenames, I wrote a filepicker program in C. Now UMENU gets the playlist's filename from the filepicker. The interface between UMENU and the filepicker consists of the contents of a temporary file. That's it! It's incredibly modular. Here's the pseudocode for how it works:
With the addition of the filepicker, my UMENU mplayer front end becomes a real application.
Steve Litt has been writing Troubleshooting Professional Magazine for ten years. Steve can be reached at his email address.

Life After Windows: The Unix Way

The creators of Unix had a vision. That vision has been carried forward by many Unix/Linux/BSD users, especially admins. The vision is the use of small executables assembled to perform larger tasks. Each small executable has been optimized to the max, and tested to perfection. Those using the vision are highly productive. It's the Unix way.

This vision is unknown and unseen by the vast majority. "Greatest Generation", "Lost Generation" and many "Boomer Generation" programmers started their careers on mainframes, using Cobol or Fortran. Applications were simpler and expected coding speeds slower. Most applications were monolithic in nature.

The Boomer and Gen-X programmers who cut their teeth on DOS know only monolithic apps written in Turbo C, Clarion, Dbase and the like. Because DOS batch files were so limited, monolithic apps were easier. This was the world in which I began programming.

The late Gen-Xers started with CGI. CGI often requires separate executables, one for each web page, but the presentation is mixed right in with the process logic. Even later Gen-Xers started with web-centric languages like PHP and ASP, or with Java applets.

The very latest adult generation, who as of yet have no name, came into the workplace after Y2K. They started with Web-centric Enterprise Java, Zope, Websphere, C# and Rails. The concept of a thick client app is foreign to most of them.

None of these generations were truly exposed to The Unix Way, even if they use Linux daily. From 1969 until the mid 1990's, exposure to Unix was a perk few enjoyed, because machine plus operating system came with a substantial five figure pricetag, or sometimes six figures. Unix jobs paid more and were quickly gobbled up with those already experienced. Thus only a small core of people were exposed to Unix in the 1970's, 1980's and early 1990's. The Unix Way was passed down from crusty veteran to adoring newbie only within this small core of Unix people. They were basically a guild.

Meanwhile, the vast majority of the programming world was using Mainframe, DEC, DOS, Windows and Web programming methods, all of which encouraged monolithic apps much more than the Unix world. The late 1990's Linux explosion produced hoardes of Linux users, but there weren't enough Unix guild types to pass on The Unix Way. As a result, programmers from the worlds of DOS, Windows and Web typically don't take advantage of The Unix Way of Application Assembly using Small Executables. In my opinion, they're missing a great opportunity.

At your next LUG meeting, look for the guy with the long gray beard. The guy who knows all the Linux commands by heart. He might speak of the good old days. He might be unemployed He might be considered by many to be irrelevent. But it's likely he was in the Unix world when Unix was the place to be. It's likely he not only knows The Unix Way -- he lives it. Learn from him -- it just might double your productivity.

Now that you know The Unix Way (Application Assembly using Small Executables), use it to create fully user interfaced apps. Use my UMENU, filepicker and recordpicker. Or make your own. Kludge a form executable or create a professional one (hopefully licensing it free software). Then, when you need a quick application, it's a matter of hours to completion. THAT'S The Unix Way.
Life After Windows is a regular Linux Productivity Magazine column, by Steve Litt, bringing you observations and tips subsequent to Troubleshooters.Com's Windows to Linux conversion.
Steve Litt is the founder and acting president of Greater Orlando Linux User Group (GoLUG).   Steve can be reached at his email address.

GNU/Linux, open source and free software

By Steve Litt
Linux is a kernel. The operating system often described as "Linux" is that kernel combined with software from many different sources. One of the most prominent, and oldest of those sources, is the GNU project.

"GNU/Linux" is probably the most accurate moniker one can give to this operating system. Please be aware that in all of Troubleshooters.Com, when I say "Linux" I really mean "GNU/Linux". I completely believe that without the GNU project, without the GNU Manifesto and the GNU/GPL license it spawned, the operating system the press calls "Linux" never would have happened.

I'm part of the press and there are times when it's easier to say "Linux" than explain to certain audiences that "GNU/Linux" is the same as what the press calls "Linux". So I abbreviate. Additionally, I abbreviate in the same way one might abbreviate the name of a multi-partner law firm. But make no mistake about it. In any article in Troubleshooting Professional Magazine, in the whole of Troubleshooters.Com, and even in the technical books I write, when I say "Linux", I mean "GNU/Linux".

There are those who think FSF is making too big a deal of this. Nothing could be farther from the truth. The GNU General Public License, combined with Richard Stallman's GNU Manifesto and the resulting GNU-GPL License, are the only reason we can enjoy this wonderful alternative to proprietary operating systems, and the only reason proprietary operating systems aren't even more flaky than they are now. 

For practical purposes, the license requirements of "free software" and "open source" are almost identical. Generally speaking, a license that complies with one complies with the other. The difference between these two is a difference in philosophy. The "free software" crowd believes the most important aspect is freedom. The "open source" crowd believes the most important aspect is the practical marketplace advantage that freedom produces.

I think they're both right. I wouldn't use the software without the freedom guaranteeing me the right to improve the software, and the guarantee that my improvements will not later be withheld from me. Freedom is essential. And so are the practical benefits. Because tens of thousands of programmers feel the way I do, huge amounts of free software/open source is available, and its quality exceeds that of most proprietary software.

In summary, I use the terms "Linux" and "GNU/Linux" interchangably, with the former being an abbreviation for the latter. I usually use the terms "free software" and "open source" interchangably, as from a licensing perspective they're very similar. Occasionally I'll prefer one or the other depending if I'm writing about freedom, or business advantage.
Steve Litt has used GNU/Linux since 1998, and written about it since 1999. Steve can be reached at his email address.

Letters to the Editor

All letters become the property of the publisher (Steve Litt), and may be edited for clarity or brevity. We especially welcome additions, clarifications, corrections or flames from vendors whose products have been reviewed in this magazine. We reserve the right to not publish letters we deem in bad taste (bad language, obscenity, hate, lewd, violence, etc.).
Submit letters to the editor to Steve Litt's email address, and be sure the subject reads "Letter to the Editor". We regret that we cannot return your letter, so please make a copy of it for future reference.

How to Submit an Article

We anticipate two to five articles per issue. We look for articles that pertain to the GNU/Linux or open source. This can be done as an essay, with humor, with a case study, or some other literary device. A Troubleshooting poem would be nice. Submissions may mention a specific product, but must be useful without the purchase of that product. Content must greatly overpower advertising. Submissions should be between 250 and 2000 words long.

Any article submitted to Linux Productivity Magazine must be licensed with the Open Publication License, which you can view at http://opencontent.org/openpub/. At your option you may elect the option to prohibit substantive modifications. However, in order to publish your article in Linux Productivity Magazine, you must decline the option to prohibit commercial use, because Linux Productivity Magazine is a commercial publication.

Obviously, you must be the copyright holder and must be legally able to so license the article. We do not currently pay for articles.

Troubleshooters.Com reserves the right to edit any submission for clarity or brevity, within the scope of the Open Publication License. If you elect to prohibit substantive modifications, we may elect to place editors notes outside of your material, or reject the submission, or send it back for modification. Any published article will include a two sentence description of the author, a hypertext link to his or her email, and a phone number if desired. Upon request, we will include a hypertext link, at the end of the magazine issue, to the author's website, providing that website meets the Troubleshooters.Com criteria for links and that the author's website first links to Troubleshooters.Com. Authors: please understand we can't place hyperlinks inside articles. If we did, only the first article would be read, and we can't place every article first.

Submissions should be emailed to Steve Litt's email address, with subject line Article Submission. The first paragraph of your message should read as follows (unless other arrangements are previously made in writing):

Copyright (c) 2003 by <your name>. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, version  Draft v1.0, 8 June 1999 (Available at http://www.troubleshooters.com/openpub04.txt/ (wordwrapped for readability at http://www.troubleshooters.com/openpub04_wrapped.txt). The latest version is presently available at  http://www.opencontent.org/openpub/).

Open Publication License Option A [ is | is not] elected, so this document [may | may not] be modified. Option B is not elected, so this material may be published for commercial purposes.

After that paragraph, write the title, text of the article, and a two sentence description of the author.

Why not Draft v1.0, 8 June 1999 OR LATER

The Open Publication License recommends using the word "or later" to describe the version of the license. That is unacceptable for Troubleshooting Professional Magazine because we do not know the provisions of that newer version, so it makes no sense to commit to it. We all hope later versions will be better, but there's always a chance that leadership will change. We cannot take the chance that the disclaimer of warranty will be dropped in a later version. 

Trademarks

All trademarks are the property of their respective owners. Troubleshooters.Com(R) is a registered trademark of Steve Litt.

URLs Mentioned in this Issue


_