README: Filepicker and Recordpicker
Copyright (C) 2007 by Steve Litt, all rights reserved
This document is free software covered by the GNU General Public License, Version 2
Version 0.0.1: December 2007
CONTENTS

Introduction

The filepicker and recordpicker (fpick and rpick) are executable binaries that form part of what some day will become a complete CLI (Command Line Interface) application assembly suite. You'll be able to build CLI mode data aware applications by gluing simple modular components together with shellscripts and simple Perl/Python/Ruby scripts.

Besides the filepicker and recordpicker components, the menu component is available today as UMENU (free software from Troubleshooters.Com), a simple persistance manager available from Troubleshooters.Com, and database interface executables such as psql, mysql and the noSQL suite. When someone writes a form builder executable for this suite, you'll be able to assemble applications much like you assemble Lego(R) bricks, using only simple shellscripts and Perl/Python/Ruby scripts. Such quick (as fast as one day for a multi-table database application) applications won't be pretty, but they'll be efficient and very fast in the hands of a touch typist.

Even without the form builder, today you can use UMENU, the formpicker and filepicker, the persistance executable, and various programs and commands available to Linux, to create simple applications in minutes.

What They Are

The filepicker and recordpicker (fpick and rpick) are executable binaries enabling the user to pick one record from a list of many. In the case of the filepicker, it picks a file from a directory, with the option of drilling down to subdirectories or emerging from directories by clicking on the double dot (..) entry. In the case of the recordpicker, it simply selects one of many records given it.

The intent of each of these programs is to provide such a thin interface that it can be quickly reliably bolted onto shellscripts via calls to it, or bolted on to Perl/Python/Ruby scripts via system() calls and the like, and bolted onto C/C++ via system() calls, fork() etc. If these programs achieve this intent, they become one of a small group of executables necessary to assemble a complete data enabled application via shellscripting or Perl/Python/Ruby. Here are the necessary executables and components:
Theoretically, armed with the preceding components, one could assemble a complex data enabled application with shellscripts and a little bit of simple logic code. Obviously, noSQL, mysql and psql are already up to the task. UMENU is also up to the task -- it's been hammered hard for over 5 years. No claim is made that the Filepicker and Recordpicker are up to the task of quick application assembly. It's my hope that over time they'll be improved to the point where they'll serve as quick and reliable building blocks. It's also my hope that someone will create the final necessary component, a formmaker.

The preceding paragraph should not be interpreted as meaning the current filepicker and recordpicker aren't incredibly useful. Using UMENU, mplayer and the filepicker, I assembled a CLI music program with a lightning quick user interface and an ultra-small memory footprint. The concept has definitely been proven.

Major Compromises

To distribute this software without spending all my time and quitting my job, I had to make many compromises. Many were small, but five are fairly substantial:
  1. The programs us a set of vi inspired keystrokes for the user interface, instead of the more intuitive arrows, pgDn, Shift+PgDn, Ctrl+PgDn, Home and End.
  2. The Recordpicker cannot call a subprogram upon choice, edit or delete, but simply returns the user's choice in a file.
#1 was caused by the fact that I didn't have time to research the C interface necessary to capture key presses as scan codes rather than characters. I was in a similar position as the original authors of vi, so I hardcoded a vi inspired set of keystrokes for navigation and choice. However, once I know how to capture keypresses as scancodes, it should not be difficult to define different interfaces as "skins", with a vi skin, and a common nav key skin with Enter being select and Esc being cancel. An Emacs skin could even be defined.

#2 was done for lack of time. Because the chosen info and user action (add, change, delete, select, cancel) are simply returned via the intermediate file, the application author must control the stack of recordpickers and formmakers. For simple apps this is no big deal. For complex apps, hopefully in the future the recordpicker will spawn formmaker scripts, which will in turn spawn recordpickers. At that point the picker/form stack will be handled automatically.

Minor Compromises

  1. CLI only: No GUI, no mouse.
  2. No hooks for Windows compatibility.
  3. Very little testing.
  4. Make file is gcc only.
#1 is actually a feature in my view. Keystrokes are by far the fastest and most reliable way to get info into most applications that aren't innately graphical. Similarly, letters and numbers are more universally understandable than little garbage cans and chain links. The fact that these components aren't graphical means they can be used in shellscripts on computers without graphics. It also means that assembling the complete application will be much easier.

#2 was for lack of time. My experience with the VimOutliner project tells me that if Windows people begin liking it, they'll submit the patches necessary to make it portable.

#3 was once again for lack of time. It works on my Mandriva 2007 box. My guess is if you have Linux with GNU gcc and the GNU make utility, it will work for you too. My guess is that if you have Unix or BSD or don't have GNU gcc and make, you'll need to do a little tweaking to make it work for you. Please keep me in the loop.

#4 is because I'm certainly not an automake expert. Indeed, this is the first C based free software I've ever distributed. I'm sure others will figure better installation methods.

Thin Interface

Both the filepicker and the recordpicker communicate via a single file. When they're called, the file contains the info necessary for them to configure themselves, and in the case of the recordpicker, the data from which to select. When they return to the caller, that file contains the user action (cancel, select, superselect, add, change, delete), and on actions except for add and cancel, it contains enough info to identify the chosen item.

Contents of Intermediate File
Filepicker Recordpicker
On entry
  • The directory to be initially searched
  • Optionally what user actions are permissible
  • Permissible user actions
  • startofdata indicator
  • The records from which to choose
On exit
  • The user action
  • On everything but cancel and delete, the full path to the file
  • The user action
  • The zero based record number of the chosen item
  • The string representation of the chosen item

The real beauty of these two executables is that they're separate executables whose only interface is the intermediate file, with absolutely no side effects. This is about the tightest encapsulation possible, converting reusability from a buzzword to a fact.

In order to facilitate multiuser/multitasking, it's the calling application's duty to use mktemp to create the intermediate file. In order to facilitate a neat hard disk, it's the calling application's duty to delete that intermediate file after reading its contents.

Copyright/License: Read Carefully

My intent is to have the CODE COMPRISING the recordpicker and filepicker be strongly copylefted free software, while the RESULTING EXECUTABLES can be used, via system() call or via a line from a shellscript, by software of any license, including proprietary. These two executables are completely standalone executables that simply query the user for a choice and report that choice back via a single intermediate file. Each is written in C. All communication from the calling (possibly proprietary) program to the picker executable is via a single intermediate file, and upon completion the picker executable returns the results to the calling program in that same
intermediate file. The sole interface between caller and callee is that one intermediate file. My executables are useful by any software and require no special library or table to exist in the caller.

I am not a lawyer, but my reading of the GNU GPL version 2 indicated to me that this usage is compatible with the GNU GPL version 2. I contacted Eben Moglen, the head lawyer for the FSF, who concurred that the usage described in the preceding paragraph is permissible under both GPL 2 and GPL 3.

So I'm licensing it GPL version 2. If for some reason it becomes problematic to use it as described in the first paragraph of this section, I'll relicense it (possibly modified BSD), but given Dr. Moglen's opinion, I seriously doubt it will ever come to that.

Installation

For the time being, installation is via a Makefile that I anticipate will work only with GNU gcc and GNU make. To find out all the possibilities, issue this command:
make
The simplest way to build it is the following:
make clean
make all
su
make install
The make all command should issue no warnings when used with GNU gcc. The make install copies the fpick and rpick executables to the /usr/local/bin directory. If you compile this on a non-Linux system or with a non-GNU C compiler, please let me know how it comes out.

User Interface Keystrokes

Time constraints prevented me from learning how to capture a keypress as a series of scan codes. Therefore, the current version cannot acquire arrow keys, page keys, home, end, and alt keys. Given that restriction, I did what the long ago author of vi did -- I made all the keystrokes letters. In fact, the letters were inspired by vi:
Navigation Keystrokes
Down one line j
Down one screen J
Down 20 screens Ctrl+J
Down to bottom G
Up one line k
Up one screen K
Up 20 screens Ctrl+K
Up to top g
   
User Action Keystrokes
Select y
Superselect Y
Cancel q
Add a
Change c
Delete d
.


If you use vi or Vim, you'll quickly get used to these keystrokes, and the interface is lightning fast.

Future versions of the picker programs will feature configurable keystroke sets so that you can have user interface "skins"; perhaps a vi skin, a CUA interface, maybe even an Emacs interface for those with flexible wrists :-). But for now, the programs are

The Filepicker

The filepicker is invoked as follows:
fpick /path/to/intermediate.file
On entry, the intermediate file must contain a line like this:
dir=/path/to/investigate
That directory is listed by fpick, subdirectories first, showing the dot and doubledot first. From there, subdirectories and superdirectories (via the doubledot) can be navigated. Optionally, the intermediate file can contain a list of permissible and forbidden user actions where user actions are one of cancel, select, superselect, add, change, delete. The superselect user action chooses a directory and returns instead of drilling down into that directory. It's used to choose directories as opposed to files.

Once the user has chosen a file, the intermediate file is rewritten to look something like this:
action=select
file=/home/jones/mp3/artiekegler.m3u
Or, if the user cancels, it looks like this:
action=cancel
There's also a superselect action to choose a directory (instead of plunging into it). The superselect action does not work on normal files -- directories only:
action=superselect
file=/home/jones/mp3
The filepicker can accommodate all useractions: cancel, select, superselect, add, change and delete. Therefore, you could create a fairly decent filemanager with a 30 line shellscript repeatedly calling the filepicker, especially if later a run user action is added.

Intermediate File Communication

The calling program's only communication with the filepicker is the intermediate file. To facilitate multiuser, multitasking operations, the calling program should create the intermediate file using mktemp or some other method to make unique filenames. The name of that intermediate file is then passed to the filepicker as arg1.

The following table shows what the calling program passes into the filepicker:

Incoming intermediate
file line		 What it does Default? Manditory?
dir=/path/directory directory in which to start choosing manditory
select=y
select=n		 SELECT user action OK? y optional
superselect=y
superselect=n		 SUPERSELECT (SELECT DIRECTORY) user action OK? n optional
cancel=y
cancel=n		 CANCEL user action OK? y optional
add=y
add=n		 ADD user action OK? n optional
change=y
change=n		 CHANGE user action OK? n optional
delete=y
delete=n		 DELETE user action OK? n optional

Notice the only manditory line is the dir= line. The defaults for the rest are perfect for applications where you're simply choosing a file, never adding, changing or deleting, and never selecting a directory.

The following lines are what the filepicker passes back through the intermediate file:

Intermediate file line from filepicker What it does When it's used
action=cancel
action=select
action=superselect
action=add
action=change
action=delete
 Informs calling program of the user's action Always written to intermediate file on normal termination.
file=/full/path/to/file Informs calling program of filename picked Written to intermediate file for select, superselect, change and delete. Not written for cancel and add.

Example Usage

fpick example
     The screenshot at the left, run in a graphical terminal, is the filepicker part of a relatively simple shellscript in which the user chooses a file to put into the less pager.

Notice the directory (/etc/) is listed on top. Note that the current record has an arrow in front of it. That arrow moves down as the user presses j and up as the user presses k.

Directories are denoted by a preliminary slash. When the user selects (by pressing y) a directory, the contents of that directory fill the list. If the user selects the double dot directory, the parent directory is listed.

The following script produced the preceding screenshot:

#!/bin/bash
TEMPFILE=mktemp
echo dir=/etc > $TEMPFILE
echo superselect=n >> $TEMPFILE
fpick $TEMPFILE
if grep -q "action=select" $TEMPFILE; then
	FN=`grep "^file=" $TEMPFILE | sed -e "s/.*=//"`;
	less $FN
else
	echo User chose to cancel
fi
    In the simple shellscript to the right, first a temporary file is created with mktemp. That's so you can run several instances of the script without them clobbering each other. Next, the starting directory is written to the temporary file, and then superselect=n is written to the temporary file, so that directories can't be selected. This is an illustration only, superselect defaults to no.

Next fpick is called to offer the user the opportunity to cruise the filesystem, starting with /etc, and choose a file to page. If the user selected rather than cancelled, the file=/etc/Muttrc that fpick wrote to the temporary file is parsed via sed to remove the file=, leaving just the full path filename, and then that file is viewed by the less pager.

The Recordpicker

The recordpicker is similar to the filepicker and uses much of the same code. However, whereas the filepicker gets its data from the filesystem, the recordpicker gets its data from the intermediate file. Whereas the filepicker sorts its data, the recordpicker does not sort, relying instead on the calling program to sort the data. Another difference is that the filepicker repeats itself as it plunges into and emerges from subdirectories, but the recordpicker returns the first time the user selects a line.

In most other respects, the two executables are very similar.

Intermediate File Communication

The following table shows what the calling program passes into the recordpicker:
Incoming intermediate
file line		 What it does Default? Manditory?
select=y
select=n		 SELECT user action OK? Y Optional
superselect=y
superselect=n		 SUPERSELECT (SELECT DIRECTORY) user action OK? N Optional
cancel=y
cancel=n		 CANCEL user action OK? Y Optional
add=y
add=n		 ADD user action OK? N Optional
change=y
change=n		 CHANGE user action OK? N Optional
delete=y
delete=n		 DELETE user action OK? N Optional
startofdata Stops search for configuration parameters and signifies that all following lines are data to go in the picklist. Manditory
Line of data Random strings of data on lines following the startofdata line.
NOTE: If there's no data, the recordpicker disables all useractions except cancel and add.

The following lines are what the recordpicker passes back through the intermediate file:

Intermediate file line from recordpicker What it does When it's used
action=cancel
action=select
action=superselect
action=add
action=change
action=delete
 Informs calling program of the user's action Always written to intermediate file on normal termination.
recno=picked_number Informs calling program of zero based line number of the chosen record Written to intermediate file for select, superselect, change and delete. Not written for cancel and add.
string=picked_string Informs the calling program of the string value of the chosen record Written to intermediate file whenever the recno= is written to the intermediate file.

Example Usage

Screenshot of the rpick example     The screenshot at the left, run in a graphical terminal, is the recordpicker part of a relatively simple shellscript presenting the user with the songs in a playlist, giving the user a chance to pick which song to play, and then playing the song.

Notice that lines are a maximum of 74 characters long, which, when added to the 3 characters used by the arrow prompt at the left of the current record, add up to 77. If a line were to go past the terminal's last column, it would screw up the picklist. It's the calling script's duty to trim all incoming lines.



Here's the shellscript that produced the preceding screenshot.

#!/bin/bash

# CONSTANTS DEFINING THE MP3 DIRECTORY AND M3U PLAYLIST TO CHOOSE WITHIN
M3UDIR=/scratch/mp3
M3UFILE=/scratch/mp3/bangertunes.m3u

# MAKE THE TEMPORARY FILE AND START THE DATA
TEMPFILE=mktemp
echo startofdata > $TEMPFILE

# CUT THE DATA TO 74 BYTES PER LINE SO THAT IT FITS ON AN 80 CHAR SCREEN
cat $M3UFILE  | cut -b1-74 >> $TEMPFILE

# PICK THE SONG
rpick $TEMPFILE

# IF USER SELECTED, PLAY IT
if grep -q "action=select" $TEMPFILE; then
	RECNO=$(grep "recno=" $TEMPFILE | sed -e "s/.*=//")

	# CONVERT 0 BASED RECNO TO 1 BASED head -n
	let RECNO=$RECNO+1

	# CHOOSE THE EXACT RECORD
	# COULD NOT HAVE CHOSEN DIRECTLY FROM STRING
	# BECAUSE STRING WAS TRUNCATED TO FIT HORIZONTALLY IN THE PICKER
	MYTEMP=$(head -n $RECNO $M3UFILE | tail -n1)

	# PREPEND DIRECTORY TO MAKE IT WORK, THEN PLAY THE SONG
	STRINGREP=$M3UDIR/$MYTEMP
	play $STRINGREP
else
	echo User chose to cancel
fi
    Notice that the calling program (this shellscript) must trim the length of each line to 74 in order to fit the 80 column terminal (including the line pointer that travels in front of the lines). Therefore, the recordpicker's returned string will not be a full filename and will not play on the play utility. Therefore, the record number must be used to reconstitute the full filename. This is very typical of most record picking operations, including those involving databases.

The shellscript first creates a unique intermediate file using mktemp. It writes the startofdata marker to the intermediate file and then all the lines of the .m3u file, suitably trimmed by the cut command.

Next it calls rpick (the recordpicker), in which the user chooses the line corresponding to the song he wants to hear. rpick writes the chosen song's zero based record number (how far it is down the list, with the top being zero), and the string representation, which in this case is meaningless because the strings have been trimmed to 74 characters, and also writes whether the user chose to select or cancel.

When the shellscript regains control from rpick, it tests the intermediate file to see if the user selected, and if so converts the recno= line to a 1's based integer, uses head and tail to grab that line out of the original .m3u file, and then plays the song.

The preceding is fairly standard use of a picklist. Most picklist apps return the chosen record's position, and that record is re-read.

Useful Future Additions

Here are some useful future additions I'd like to make to the pickers:

Read scancodes on keypresses

The current keyboard interface is fast and clean, but it's completely inobvious. Requiring a user to read the manual in order to use the software is sooooo 1992. Once I can read keypresses as scancodes, all keyboard combinations can be recognized, and the keyboard interface can be based on intuitive cursor keys, page keys and the like.

Here are some web pages devoted to scan codes:

User interface keyboard "skins"

Reentry at same record number and screen position

Other user actions including VIEW and RUNRead scancodes on keypresses

Configurable line truncation on the filepicker



Application Assembly By Shellscript: More Components

The end goal is to make it easy for a beginning program, sysadmin or even a "power user" to assemble an application, possibly a database application, from executable components like fpick and rpick. They'll assemble these applications much like you would assemble a model building using Lego(R) bricks. Other components available today are:
Other executables will be written as they become necessary. One example is a simple stack handler, which is necessary to stack layers of picklists and forms in a data app. This is very simple to create, and could be done with a simple shellscript, or if that proves too slow, a simple C program. As people develop more and more

One necessary executable component is unwritten and will be difficult to write. It's the form builder. I anticipate that until this project gains critical mass, forms will be serviced by kludges built from Vim, rotating prompts (1970's technology but it worked), dialog, cursor, and various Linux utilities I no longer remember. That's OK -- once the value of this concept is realized, somebody will quickly build a real form builder executable.

Trademarks

Lego is a trademark of The Lego Group.