Troubleshooters.Com®, Linux Library, and DIY Linux Present:

Installing Suckless Init/Felker Init Plus Daemontools-Encore on Plop Linux

Copyright © 2015 by Steve Litt

Contents:

Welcome!

You're DIY. If you weren't DIY, you wouldn't be reading this. This material isn't easy, and it isn't something you do in an afternoon.

If others find out you installed Suckless Init on Plop Linux, many will ask you why you're wasting your time. Try as you might, you'll never get them to understand why you care about what's under the hood. Their exclusive focus is learning the latest cockpit controls --- leave what's under the hood for the developers.

That's OK, you're DIY and you've heard it all before. You're DIY and you like it that way. And you like the DIY side benefit: You're one of the vanishing breed who can systematically troubleshoot. When their trial and error and lookup and ask and speculation, guesswork and prayer don't work, and the answer isn't on linuxquestions.org or stackoverflow.com, the cockpit control aficionados come to you to troubleshoot the problem. Under the hood. And you always get your bug.

When you've reproduced the work in this document, you'll know the exact basics of an init process. You'll have a pretty good idea how to configure or troubleshoot any init process. You built one up from scratch. To get any more down and dirty you would have had to code it yourself in C.

So welcome aboard and buckle up. It's going to be a long, wild ride, and when it's done, in most init discussions, and more than a few systems-programming discussions, you'll be the smartest guy in the room.

Introduction

Here's the overview of the init system you'll build in this document:

Overview of the init system

That's it. If you understand any daemontools like program, and you understand a 16 line init program that basically forks off a shellscript and then kills zombies, you understand this init system.

Of all the workable init systems for Linux, Suckless Init is probably the simplest (except for Felker Init, which is discussed at the end of this document). As a matter of fact, Suckless Init (and Felker Init) cannot even manage daemons, so it must pass that task on to a daemon manager like daemontools-encore. As previously mentioned, this document walks you through installing Suckless Init with daemontools-encore on a VirtualBox VM with Plop Linux installed. Plop Linux is one of the simplest Linux distros available. To work with this document, you need to be familiar with the following prerequesites:

I'd recommend performing this set of tasks on a VirtualBox VM. That's how I did it, it worked perfectly, and iterative experimentation was very fast. Besides VirtualBox, you'll need to download the following software:

Plop Linux comes stock initted by Sysvinit (the one with /etc/inittab and all the run scripts in /etc/rc.d/init.d with symlinks in /etc/rc.d/rc3.d). This document walks you through replacing Sysvinit, by the combination of Suckless Init plus Daemontools Encore, in discreet steps.

Install Plop Linux

Plop Linux is so simple it's perfect for experimentation and proofs of concept. That's why I used it as the Linux distro for this document. To install Plop Linux on a VirtualBox VM, follow these instructions:.

Daemontools Overview

Daemontools is so good and so unusual that many people have written process managers and even whole init systems that are supersets of Daemontools. Daemontools-encore is one such superset: a very modest superset. If you know daemontools, you pretty much know daemontools-encore, and vice versa. For the rest of this section, I'll just use the world "daemontools".

Daemontools manages processes. That's all it does. When a process dies, daemontools restarts it. The default way to use daemontools is to use a directory called /service. If a new direct-under-root directory violates your principles, you can put it anywhere on the root partition, even under /etc, but for the purpose of this document it's /service. Actually, it doesn't need to be under the root partition, it could be in a mounted partition, but then that partition must be mounted before starting daemontools.

Daemontools basically follows this pseudocode algorithm:

forever:
 foreach symlinked dir under /service:
  if its service isn't running:
   exec its run script to make it run

To make a new service, make a directory, not in the /service tree, somewhere that's mounted before daemontools starts. Let's call the new directory /root/sshd.

Note:

In reality, most folks put such directories under a common parent directory. In this document, I use /var/service as that parent. Having them all under the common parent makes things conceptually easier, and less error prone. The parent must be mounted and accessible at all times when daemontools is running. Which means that if the parent is on a non-root partition, that partition must be mounted before your calls to lk_prepare and svscanboot.

Anyway, back to the generic service creation description: Within /root/sshd, create a shellscript, executable by root, that exec's sshd in the foreground. So it would look like the following:

#!/bin/sh
exec /usr/sbin/sshd -d

In the preceding run script, the -d stands for "daemon" and means that sshd stays in the foreground, which is what you want for all daemontools daemons. Note that this run script has no code for checking that processes it depends on, like the network, are already running. Such dependency checking is simple and discussed later in this document.

Anyway, once you've finished the run script, you add it to daemontools with a simple symlink command:

ln -s /root/sshd /service/sshd

Once you do that, daemontools runs sshd.

The command to run daemontools is svscanboot. The following outline shows the command tree for daemontools:

Here's a ps axf session, which is too wide for some handhelds, showing my daily driver desktop, where daemontools manages a few processes:

A few things to note about the preceding ps output:

Daemontools starts up its processes in no particular order. Many times this is a good thing, but occasionally it makes life difficult. So I wrote a set of shellscripts called LittKit that make it easy to specify the order daemontools runs its process in, when daemontools first runs.

One more thing about daemontools: It makes huge use of the Unix filesystem. Most of your daemon config is done via files and directories. There are no configuration files: Just files and directories. For instance, a file called down within a service directory tells daemontools not to start that service. The more you work with it, the more sense this makes.

Daemontools has many commands, but two are very useful to the user (or admin):

You can learn more about these two commands in other documents.

Here are some good documents to add a little more info:

  1. Daemontools Intro.
  2. daemontools, by djb.
  3. daemontools-encore, by Bruce Guenter.

To recap, here are some facts:

Install Daemontools-Encore

Download daemontools-encore from http://untroubled.org/daemontools-encore/. It is a tarball, a .tar.gz. Untar it, and follow the instructions in the README file inside the directory made by the untar. Your mileage may differ, but I didn't need to change any of the conf-* files, including the conf-bin or the conf-man. In my case, all I needed was make, followed by make install performed as user root.

For me, the only immediate evidence that the installation succeeded was that addition of several files to /usr/local/bin, including svscanboot, svscan, supervise, svstat, and svc.

Testing Daemontools-Encore

You need to test that daemontools-encore really works. This is a several-step process.

Create a test program to be daemonized

First, create and permission executable by all the following /usr/local/bin/dolog.sh:

#!/bin/sh

date >> /tmp/junk.log
sleep 10

date >> /tmp/junk.log
sleep 10

date >> /tmp/junk.log
sleep 10

date >> /tmp/junk.log
sleep 10

date >> /tmp/junk.log
sleep 10

date >> /tmp/junk.log
sleep 10

The preceding shellscript is the program to daemonize. It is on the path, and writes /tmp/junk.log every ten seconds, six times. It runs a total of a minute, updating /tmp/junk.log every ten seconds. When running, its progress can be monitored, on a separate terminal, with the following command:

tail -fn0 /tmp/junk.log

Run this program, and make sure it writes /tmp/junk.log six times and then quits.

Because this program runs for a minute and stops, it tests not only daemontools-encore's ability to start it, but daemontools-encore's ability to respawn it after it dies.

Build the Proof of Concept Service

  1. Log in as user root.
  2. mkdir /service
  3. mkdir /var/service
  4. mkdir /var/service/dolog
  5. Create the following /var/service/dolog/run: 
    #!/bin/sh
echo Starting dolog
exec /usr/local/bin/dolog.sh
  6. chmod u+x /var/service/dolog/run
  7. ln -s /var/service/dolog/ /service/dolog
  8. /usr/local/bin/svscanboot

Test the Proof of Concept Service

If everything's gone as planned, /usr/local/bin/dolog.sh is running, and every sixty seconds, after it terminates, it gets run again. Here's how to test:

tail -fn0 /tmp/junk.log

Expected output is a new line with a time ten seconds than the last, every ten seconds. However, it might occasionally skip more than 10 seconds. This corresponds to times when dolog.sh terminates and must be restarted by daemontools.

For a more complete test, create the following testjunk.sh and make it executable:

#!/bin/sh

while read theline; do
  echo -n "$theline %%% "
  svstat /service/dolog
done

Now, as user root (or else the following command partially errors out), run the following command:

tail -fn0 /tmp/junk.log | ./testjunk.sh

The preceding shows you, upon each and every log file output, the log entry, which consists of the time, followed on the same line by information about the service. You'll notice that one out of every six outputs, the PID changes and the uptime drops back down to zero, proving that daemontools is respawning it. If you get output like this text, which is too wide for some handhelds, you're good:

If at first it doesn't work, keep troubleshooting until it does. Keep referring to my Daemontools Intro for troubleshooting ideas, and remember, the ps command is your friend.

Enable Daemontools-Encore to Start at Boot

Being a process/daemon manager, the time for daemontools-encore to start is at boot, after everything else has been done. Because Plop Linux natively uses Sysvinit with /etc/inittab, the best way to have daemontools-encore start at boot is to put the following line as the final executable line of /etc/inittab.

Once you've done that and saved /etc/inittab, reboot the VM, run the same tests, and see if daemontools-encore continues to run the dolog service.

SV:12345:respawn:/usr/local/bin/svscanboot

Once you've done that and saved /etc/inittab, reboot the VM, run the same tests, and see if daemontools-encore continues to run the dolog service. Troubleshoot as necessary, starting with determining whether the ps command shows a process called svscanboot.

Note:

Once you've outgrown the usefulness of the dolog daemon, you can prevent it from automatically starting, and kill the currently running copy, with these three commands:

  1. touch /var/service/dolog/reallydown
  2. touch /var/service/dolog/down
  3. svc -dx /var/service/dolog

Note:

Once you've made /service/dolog a symlink to /var/service/dolog, /service/dolog and /var/service/dolog are different names for the same directory, so that changes you make in one get identically changed in the other.

Transferring Processes From Sysvinit to Daemontools-Encore

This section is theory. The actual procedures are enumerated in later sections.

Plop Linux ships with the Sysvinit init system, and Sysvinit does a perfectly fine job of managing processes, at least in the unchallenging conditions of Plop Linux. Because you'll be replacing Sysvinit with Suckless Init, which does not manage processes at all, you'll need to have daemontools-encore manage all the processes formerly managed by Sysvinit. Because daemontools-encore (and most daemontools inspired software) doesn't straightforwardly allow boot time ordering of processes, and doesn't facilitate run-once type processes, you'll be adding my LittKit daemontools enhancement scripts to easily add that capability to daemontools-encore.

The fact that Plop comes from the factory with sysvinit booting means we can deduce what the init process needs to do by looking at the /etc/inittab and the contents of /etc/rc.d/rc3.d. Those resources will tell you what processes need to be run, in what order, and to some extent give you a clue as to whether the processes should be respawn-managed or run-once.

You might despair while looking at the contents of /etc/rc.d/rc3.d. Complex, ugly, and very sysvinit-centric. It's not as bad as it seems, because you don't need to make an init process that works on every possible computer: You just need to make one that works on yours. Just as one example, /etc/rc.d/rc3.d/S20network must loop through all sorts of stuff to find out the name of the network adapter. But you know, from running Plop, that the network adapter is named eth0, so you can hardcode that.

Another example: /etc/rc.d/rc3.d/S81cups is hopelessly convoluted, at 225 lines, as it ships from the Plop factory, because it needs to work with any possible computer, and probably any possible distro. That gigantic mess becomes the following daemontools-encore run script:

#!/bin/sh

# NO USE RUNNING CUPS UNTIL THE NETWORK IS FUNCTIONAL
if ! ping  -c1 -W1  10.0.2.2 1>&2 > /dev/null; then
	# NETWORK NOT UP, WAIT AWHILE AND EXIT.
	# svscan will try again after run script exits.
	# 10 seconds is a good interval to wait
	# for the network to come up.
	sleep 10
	exit 1
fi

exec /opt/sbin/cupsd -f -c /opt/etc/cups/cupsd.conf

The preceding command is an adaptation of the command, that actually does the job, in the 225 line original /etc/rc.d/rc3.d/S81cups. The -f option keeps the program running in the foreground, which is how you run all services administered by daemontools inspired software.

Warning!

As the cupsd command ships in Plop's /etc/rc.d/rc3.d/S81cups script, the cupsd command uses an uppercase -C option, which is undocumented, instead of the lower case -c option, which means "use this config file". The upper case option causes daemontools-encore to keep spawning more and more cupsd instances, gobbling up your RAM and more importantly, preventing you from logging into http://127.0.0.1:631. If you can't log in, check the case of that parameter.

Note:

The if statement inside the run script tests for the network being up. The common ways to enforce process dependencies in daemontools-inspired programs is to test for the dependency being ready, and if it's not, sleep and exit, knowing that daemontools will retry within five seconds of the exit. Note that the 10.0.2.2 is the default address for VirtualBox VMs. If this is on bare metal, you'll need a different test to make sure the network's ready.

Notice that without the dependency checking, the run script is only two lines long. A lot different from the 225 line original.

Look at /etc/inittab. The only processes it really spawns are the virtual terminals (gettys): The agetty commands.

Now look at the /etc/rc.d/rc3.d startup scripts:

  1. S10sysklogd
  2. S20network
  3. S50lrc.local
  4. S81cups

Those four are spawned in that order. Note that S50lrc.local is really a script, meant to do lots of bootup-introductory things. It's probably not meant to respawn: It's probably a run-once. Sysklogd and network spawn before it, Cups spawns after it. The Virtual Terminals certainly can spawn after it, although maybe they can spawn before it too.

The trick is to transfer all of this to management by daemontools-encore. We'll do it a step at a time...

Transferring the Virtual Terminals to Daemontools-Encore

Virtual terminals are the non-GUI terminals you get on bare metal Linux when you press Ctrl+Alt+F3 or whatever. On VM's, you go to them when you press RightCtrl+F3 or whatever, assuming that RightCtrl is the key used to remove focus from the VM. The official name for such a key is the "host key". Anyway, Virtual Terminals on Linux are normally run by respawning the agetty program.

Take a look at /etc/inittab:

# Begin /etc/inittab 
 
id:3:initdefault:

si::sysinit:/etc/rc.d/init.d/rc sysinit 
 
l0:0:wait:/etc/rc.d/init.d/rc 0 
l1:S1:wait:/etc/rc.d/init.d/rc 1 
l2:2:wait:/etc/rc.d/init.d/rc 2 
l3:3:wait:/etc/rc.d/init.d/rc 3 
l4:4:wait:/etc/rc.d/init.d/rc 4 
l5:5:wait:/etc/rc.d/init.d/rc 5 
l6:6:wait:/etc/rc.d/init.d/rc 6 
 
ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now 
 
su:S016:once:/sbin/sulogin 
 
1:2345:respawn:/sbin/agetty --noclear tty1 9600 -a root
2:2345:respawn:/sbin/agetty tty2 9600 
3:2345:respawn:/sbin/agetty tty3 9600 
4:2345:respawn:/sbin/agetty tty4 9600 
5:2345:respawn:/sbin/agetty tty5 9600 
6:2345:respawn:/sbin/agetty tty6 9600 
 

SV:12345:respawn:/usr/local/bin/svscanboot

# End /etc/inittab

The only things from /etc/inittab that need transferring to daemontools-encore are the calls to /sbin/agetty.

Let's start by replacing one virtual terminal (tty). Specifically, tty2. Comment it out of /etc/inittab, and perform the following steps to get it set up with daemontools-encore:

  1. mkdir /var/service/tty2
  2. Create the following /var/service/tty2/run program: 
    #!/bin/bash
echo Starting tty2
exec /sbin/agetty tty2 9600
  3. chmod u+x /var/service/tty2/run
  4. ln -s /var/service/tty2 /service/tty2
  5. Reboot the VM
  6. Press Right-Ctrl+F2 (or Ctrl+Alt+F2 if bare metal) to get to tty2
  7. Log in, do a few things, log out.
  8. Repeat the preceding step once more.
  9. If you see a "no job control on this shell" message on logging in, you probably forgot the exec in your run script. Add it to clear the message.

Once you've gotten Virtual Terminal 2 working, do the same thing with 3 through 6. Remember to comment them out of /etc/inittab as well as putting them in daemontools-encore. Reboot, and you should have Virtual Terminal 1 still controlled by Sysvinit, but 2 through 6 controlled by daemontools-encore and working well.

Transferring Cups to Daemontools-Encore

One look at /etc/rc.d/rc3.d/S81cups can make you a pessimist. But you have an advantage with the following command:

[root@ploplinux ~]# ps ax | grep cupsd | grep -v grep
 2879 ?   Ss 0:00 /opt/sbin/cupsd -f -C /opt/etc/cups/cupsd.conf
[root@ploplinux ~]#

The preceding is basically the command you use in your daemontools-encore run script, although you'll use a lowercase -c rather than the uppercase. Do the following as user root:

  1. /etc/rc.d/rc3.d/S81cups stop
    • This stops Cups.
  2. ps ax | grep cupsd | grep -v grep
    • The preceding command should show no output if you've really stopped Cups. If it shows output, investigate before going on. You don't want multiple instances of Cups.
  3. mkdir /etc/rc.d/rc3.d/commentout
  4. mv /etc/rc.d/rc3.d/S81cups /etc/rc.d/rc3.d/commentout
  5. The preceding steps prevent from sysvinit from starting Cups again, at least in runlevel 3. Now you need to make daemontools-encore start and manage Cups...
  6. cd /var/service
  7. mkdir cupsd
  8. Create the following run script within /var/service/cupsd: 
    #!/bin/sh

# NO USE RUNNING CUPS UNTIL THE NETWORK IS FUNCTIONAL
if ! ping  -c1 -W1  10.0.2.2 1>&2 > /dev/null; then
	# NETWORK NOT UP, WAIT AWHILE AND EXIT.
	# svscan will try again after run script exits.
	# 10 seconds is a good interval to wait
	# for the network to come up.
	sleep 10
	exit 1
fi

exec /opt/sbin/cupsd -f -c /opt/etc/cups/cupsd.conf
  9. In the ping of the run command shown in the preceding step, use whatever IP address should ping if the network is up. 10.0.2.2 is the default address for the host, as seen by a guest, on VMs with standard network configuration.
  10. In the run command, make sure to use the lowercase -c option rather than the uppercase -C option. Otherwise, you'll appear to have hard to diagnose daemontools-encore problems.
  11. chmod -u+x /var/service/cupsd/run
  12. ln -s /var/service/cupsd /service/cupsd

The preceding steps stop Cups, prevent sysvinit from starting Cups again at runlevel 3, enable daemontools-encore to run and manage Cups, and actually start Cups.

cd /var/service

To check whether you've succeeded, perform the following command. Your output should have one supervise cupsd and one /opt/sbin/cupsd, just as shown in this output, which is too wide for some handhelds.

Once you see exactly one of each of those two commands, you can use Firefox to test Cups at 127.0.0.1:631. Be sure to try to create a new printer, to make sure Cups let's you log in as user root.

Danger !!!

Log into Cups as root, never as a lesser user. The lesser user will be refused, and Firefox won't ask you for another username and password. I had to subsequently use Chromium for web access to my local Cups server after trying to log into Cups as user slitt.

Introduction to LittKit

This section is all theory. Application will be covered in later sections.

We've gone as far as we can go without introducing LittKit. LittKit is a collection of shellscripts that makes startup ordering and run-once processes easy in daemontools-encore, probably daemontools itself, and likely in most daemontools-inspired software. You're going to need startup ordering and run-once in order to use daemontools-encore as the process management part of an init.

When djb created daemontools, I'm pretty sure his main purpose was to run his daemons like djbdns and qmail. I doubt he was envisioning daemontools as being part of system startup, so, to keep things simple, he didn't include these two features:

By the way, when I say "run-once capability", I mean to run the process once, and if or when the process terminates, don't do anything more. Daemontools' innate behavior is to rerun any of its managed processes that have stopped. Some daemontools-inspired software, and I think that includes daemontools-encore, have additional capabilities that do enable native run-once processing.

Theoretically, daemontools and its ideological descendents don't need startup ordering. Startup ordering could be handling with dependencies, like the if statement you remember from the Cups run script. There are two problems with handling order as dependencies:

Using dependencies this way, on bootup of any kind of substantial system, is theoretically possible but too problematic in the real world. Fortunately, there's another way to invoke startup ordering in daemontools and its friends: By installing and removing down files, in each process' directory, at just the right time. This method has four disadvantages:

So to fix these problems, I made a set of shellscripts called LittKit, which address the ordering problems in the following ways...

LittKit consists of the following shellscripts, all of which should be on the executable path:

So the boot goes something like this:

  1. The simple init program, which doesn't have its own process management, runs and hands off to a shellscript, perhaps /bin/rc.init.
  2. /bin/rc.init runs anything necessary for daemontools to work (path, making hard disk available, etc.)
  3. /bin/rc.init then runs lk_prepare to prepare daemontools for ordered startup. lk_prepare puts down files in every symlink directory under /service unless the symlink directory already has a nodown file. Only one of the
  4. /bin/rc.init exec's daemontools' svscanboot program, which henceforth has the same PID as /bin/rc.init did.
  5. svscanboot runs svscan
  6. svscan polls each symlink directory under /service, running any that don't have a down file.

Adminning a daemontools-inspired system equipped with LittKit would be almost the same as without LittKit. Here are the differences:

So much for theory. Let's start converting to LittKit...

Beginning to Work With LittKit

Like everything else, introducing yourself to LittKit is easier before you switch the PID1 part of your init. This section starts you off gently.

Installing LittKit is merely a case of copying lk_* from your LittKit distribution to a directory on your computer that's part of the executable path. Be careful though: Just because it's part of the executable path in the steady state does not mean it's on the executable path early in the boot. This document assumes your lk_* files are in /usr/local/bin.

Before beginning this exercise, it's a good idea to get a look at the pre-LittKit baseline. If you've been following along with the exercises in this document, you have Cups and tty2 through tty6 started by daemontools. Here are the relevant parts of your ps axjf command's output, which are too wide for some handhelds.

The important thing to notice is that svscanboot, whose parent's PID is 1, spawns svscan. The svscan executable, whose parent is svscanboot, spawns all the supervise processes. The supervise processes spawn the actual process like agetty and cupsd.

Now let's make a slight change. Create the following /usr/local/bin/rc.littkit:

#!/bin/sh
exec /usr/local/bin/svscanboot

Make sure /usr/local/bin/rc.littkit is executable by user root.

Next, in /etc/inittab, change the reference to svscanboot to rc.littkit by copying, changing, and commenting out, as shown in the following example from /etc/inittab:

#SV:12345:respawn: /usr/local/bin/svscanboot
SV:12345:respawn: /usr/local/bin/rc.littkit

What you've done is create a /usr/local/bin/rc.littkit that does nothing but exec svscanboot. Because exec simply runs the called process in the current process ID, theoretically, after booting this, ps axjf should still show svscanboot directly descended from PID1, and all the rest descended directly or indirectly from it.

Reboot now. If you've done everything right, it should boot just like before, and the relevant output of ps axjf should be the same, with svscanboot directly descended from PID1, and the ancestor of all the ttys and Cups.

Once you have that running correctly, you'll add two lines just before the exec in /usr/local/bin/rc.littkit. The revised file appears as follows:

#!/bin/sh
export PATH=$PATH:/usr/local/bin
lk_prepare /service
exec /usr/local/bin/svscanboot

You can see the PATH and lk_prepare lines have been added. I needed to add the PATH line because, at that early point of the boot, /usr/local/bin was not on the executable path. The lk_prepare line writes down files to everything direct subdirectory of /service that doesn't contain an explicit nodown file. Because you haven't yet put a nodown file anywhere, the effect is to prevent daemontools from starting up Cups or the virtual terminals (ttys). That's not a problem, though, because /etc/inittab still directs sysvinit to run tty1.

Now that you've added those two lines to /usr/local/bin/rc.littkit, reboot. Verify that ps axjf shows an absence of the agetty and cupsd processes, although the supervise processes are all there. This is what a down file does: The directory still gets supervised, but the actual command is not spawned.

The next step is to explore nodown files. Perform the following:

touch /service/tty2/nodown
touch /service/cupsd/nodown

Reboot, run ps axjf, and note that now cupsd and the getty executable for tty2 are running, and you can switch to Virtual Terminal 2. This is what the nodown file does for you: It enables a service to run even after you've run lt_prepare. If things are running as described in this paragraph, you've just proven the concept of nodown files.

Ordered Startup Hello World

In this subsection you'll do an ordered startup of daemontools-encore managed processes, even though sysvinit orders processes just fine. The purpose is to make sure you can order things before you switch to the much simpler Suckless Init.

First, delete the nodown files you put in /service/tty2 and /service/cupsd. Next, created the following /root/dtinit.sh file and make it executable by user root:

#!/bin/sh
initialsleep=12
sleepafter=6

sleep $initialsleep
lk_runsvc /service/tty2 $sleepafter
lk_runsvc /service/tty3 $sleepafter
lk_runsvc /service/tty4 $sleepafter
lk_runsvc /service/tty5 $sleepafter
lk_runsvc /service/tty6 $sleepafter
lk_runsvc /service/cupsd $sleepafter
lk_forever 10

In order to present you with observable diagnostic output, the preceding script starts the ttys in numerical order, six seconds apart, and then starts cupsd, and then goes into an infinite sleep loop (each sleep is specified to be 10 seconds). The last argument on the lk_runsvc calls is optional: It's the number of seconds to sleep after starting the service. It defaults to 1 second, and you can specify it as 0 for quicker boots if that makes sense. The reason for the initial sleep 12 is to give you a chance to log in and run a diagnostic shellscript. Obviously, the sleep 12 should be removed in production, and the "sleepafter" times of the ttys should be 1 or 0 in production.

When all is said and done, the /root/dtinit.sh script will be your main init script, and it will be executed by daemontools as the earliest-started daemontools service. It will fire off all the other daemontools services, in the desired order.

The reason for the lk_forever call at the end of /root/dtinit.sh is to have it simulate a daemon rather than a simple script. This fools daemontools into not restarting it every five seconds, so it runs just once. Keep in mind, with most newer daemontools-like implentations, such as daemontools-encore, there are less kludgy ways to achieve a run-once.

Anyway, for the time being, right now, /root/dtinit.sh is a simple diagnostic script.

The Diagnostic

In order to ascertain that it's doing what it's supposed to, you need to see the relevant ps axjf output relevant to daemontools-encore. Here's how you do it. First, create the following fromuntil.awk somewhere on your executable path, make it executable by all:

#!/usr/bin/awk -We

# PUBLIC DOMAIN, NO WARRANTY

function usage(args){
   printf "Error: ARGC was %d, should ", args
   print "have been 2.\n\n"
   print "USAGE: cat file | fromuntil.awk "
   print "regex_start regex_ignore\n"
   print "or     upto.awk regex_start "
   print "regex_ignore < file\n"
   exit 1
   }
BEGIN{
   if((ARGC < 2)||(ARGC > 3)){
		usage(ARGC)
   }
   found=0
   alltheway = 1

   start = ARGV[1]
   if(ARGC == 3){
      ignorefrom = ARGV[2]
	   alltheway = 0
      }
   else {
      ignorefrom = "infinity"
   }

   ARGC = 0
   }



$0 ~  start {
   if(found == 0) found = 1
   }
$0 ~  ignorefrom {
   if((found == 1) && (alltheway == 0)) {
      found = 2
      }
   }

found == 1 {
   print $0
   }

Next, put the following show_dt_procs.sh on your executable path, make it executable for all:

#!/bin/sh
while true; do
  echo; echo; echo
  rm -f junk.jnk
  ps axjf > junk.jnk
  cat junk.jnk | grep -v readproctitle | \
     fromuntil.awk svscanboot readproctitle
  rm -f junk.jnk
  sleep 2
done

Why the Kludge?

Why did the preceding code redirect ps axjf into junk.jnk, and parse junk.jnk? Why not simply pipe the output of ps axjf into fromuntil.awk?

The answer is that on Plop Linux in a VirtualBox VM, the direct pipe produces no output, for reasons I don't have time to fully investigate. Hence the kludge.

Finally, on the executable path, create and make executable by all the following file called jjj

#!/bin/sh
/root/dtinit.sh &
show_dt_procs.sh

jjj is what you'll run the instant you reboot, in order to observe the startup created by /root/dtinit.sh.

Test The Hello World

Once again, make sure you've deleted all the nodown files, and reboot. You should emerge from the reboot with no ttys spawned by daemontools (remember, tty1 is still spawned by sysvinit), and no cupsd spawned by daemontools. To test that, run the following command:

test_dt_procs.sh

The preceding should, over an extended period (as long as you'd like, actually), show svscanboot spawning svscan, which spawns supervise for tty2-6 and the cupsd service, but those supervise commands spawn nothing, so cupsd and the ttys are not really running. This is LittKit performing as designed: lk_prepare put down files in all the directories without noboot files, and you previously deleted all the noboot files, didn't you?

Now watch what happens as the ordered startup specified by /root/dtinit.sh is run. Perform the following command:

jjj

If all is good, you'll start out seeing no supervise processes spawning anything, and then 12 seconds later the supervise process for tty2 spawns agetty, and then every 6 seconds the next numerically higher tty spawns its agetty, and last but not least, the supervise for cupsd specifies its cupsd process.

Once you get things to work as in the preceding paragraph, you've proven your ability to perform an ordered start on daemontools-encore. But wait, there's more.

Initting From Daemontools Itself

The proof of concept was nice, but what you really want is for daemontools-encore to run the ordering script itself!

The solution's conceptually simple:

  1. Create a daemontools-encore service called dtinit. It could be called anything, this is just a name I've chosen.
  2. Put a nodown file in this new service's directory.
  3. Have the new service's run script exec /root/dtinit.sh

Now let's do it:

  1. Edit /usr/local/bin/jjj and remove the line referencing /root/dtinit. That will be done automatically from now on.
  2. mkdir /var/service/dtinit
  3. cd /var/service/dtinit
  4. touch nodown
  5. touch down . This prevents the service from "coming alive" before rebooting, but will have no effect at all once there's been a reboot.
  6. Create and make executable by root the following run script: 
    #!/bin/sh

exec /root/dtinit.sh
  7. ln -s /var/service/dtinit /service/dtinit
  8. reboot
  9. Immediately after reboot run jjj and watch what happens.

Now, just for confirmation, perform the actions in the following session, and compare your results:

[root@ploplinux ~]# cd /service
[root@ploplinux service]# svstat *
cupsd: up (pid 2170) 256 seconds, running
dtinit: up (pid 1797) 348 seconds, running
tty2: up (pid 1857) 336 seconds, running
tty3: up (pid 1921) 320 seconds, running
tty4: up (pid 1978) 304 seconds, running
tty5: up (pid 2042) 288 seconds, running
tty6: up (pid 2106) 272 seconds, running
[root@ploplinux service]#

In the preceding, notice that all services are up and running. Notice that the service that's been up for the longest is dtinit, and it's 12 seconds older than service tty2, which made sense because in my /root/dtinit.sh, I had set my $initialsleep variable to 12 seconds and my $sleepafter variable to 16 seconds. Notice in the preceding session printout, the intervals correspond to my $initialsleep and $sleepafter variables, and their age is in descending order of their calls in /root/dtinit.sh. This is proof that daemontools-encore ordered its own startup, with a little help from LittKit.

What If It Didn't Work?

Stuff happens. Things go wrong. If the preceding didn't work, start by making sure you really followed the instructions. Is svscanboot really running, and is there only one of them? Is /service/dtinit a symlink to /var/service/dtinit? Is /var/service/dtinit/run executable, and does it end with an exec statement? Did you remove the call to /root/dtinit.sh from /usr/local/bin/jjj?

Beyond the preceding paragraph, you'll always troubleshoot productively if you believe the Troubleshooter's Philosophy and consistently perform the Troubleshooter's Mantra after every diagnostic test.

Troubleshooter's Philosophy

Don't try to fix it, just try to narrow it down.

Troubleshooter's Mantra

How can I narrow it down just one more time?

The Troubleshooter's Philosophy and Troubleshooter's Mantra are right out of my instructional materials on Troubleshooting, and they work wonders.

Here are a few references that might be helpful in troubleshooting daemontools-encore and related problems:

LittKit Lessons Learned

LittKit consists of several shellscripts and a couple reserved filenames (nodown and reallydown). LittKit is syntactic sugar to make hacking of down files, for the purpose of controlling daemontools startup order, appear less kludgy.

You've now used LittKit to have daemontools-encore control its own startup order. This is very helpful, if not vital, in a bootup/init situation. You'll see that when we finally substitute Suckless Init for Sysvinit on Plop Linux.

Network Dependencies

Many, perhaps most of the services run by daemontools-encore, require the network to be up. Cups and sshd are just a couple. This is so common that, to save time and writing in various run files, you'll make a program called netisdown, so within your span.run scripts you can simply do the following: 

#!/bin/sh
if netisdown; then
  sleep 10
  exit 1
fi

exec /opt/sbin/cupsd -f -c /opt/etc/cups/cupsd.conf

The following is the code for /usr/sbin/netisdown:

#!/bin/sh
ping -W 1 -w 1 -c1 10.0.2.2 > /dev/null
rtrn=$?

if test "$rtrn" = "0"; then
  rtrn=1
elif test "$rtrn" = "1"; then
  rtrn=0
else       ### SHOULD NEVER HAPPEN
  rtrn=0   ### SOMETHING MUST BE WRONG, SAY ITS DOWN.
fi
exit $rtrn

The netisdown program will be used a great many times throughout this document.

Transferring From S50lrc.local

Now that we have a self-ordering daemontools-encore, let's transfer some more processes and commands from the infamous /etc/rc.d/rc3.d/S50lrc.local to daemontools-encore. For the most part, we'll start from the end of the file and work back. First subject, the call to alsactl restore.

  1. Comment out the alsactl restore line from /etc/rc.d/rc3.d/S50lrc.local Use four pound signs so that you can tell what you commented out from the original comments.
  2. mkdir /var/service/alsactl_r
  3. cd /var/service/alsactl_r
  4. Create and make executable by root the following run file: 
    #!/bin/sh
/opt/sbin/alsactl restore
lk_forever 3600
  5. chmod u+x run
  6. touch down
  7. Add lk_runsvc /service/alsactl_r $sleepafter to /root/dtinit.sh, just before the cupsdline.
  8. ln -s /var/service/alsactl_r /service/alsactl_r
  9. Reboot and check

By the way, this might be a good time to crank down the $initialsleep in /root/dtinit.sh to 4, and the $sleepafter to 2. Eventually you'll hand craft the file and make most of them either 1 or 0, but for the time being, try 4 and 2.

Acpid

Now do the same for acpid:

  1. Comment out the aspid line from /etc/rc.d/rc3.d/S50lrc.local Use four pound signs so that you can tell what you commented out from the original comments.
  2. mkdir /var/service/acpid
  3. cd /var/service/acpid
  4. Create and make executable by root the following run file: 
    #!/bin/sh
exec /usr/sbin/acpid -f
  5. chmod u+x run
  6. touch down
  7. Add lk_runsvc /service/acpid $sleepafter to /root/dtinit.sh, just before the alsactl_rline.
  8. ln -s /var/service/acpid /service/acpid
  9. Reboot and check

Danger!

Notice the -f on the end of the run command. That -f runs acpid in the foreground, which is how all daemontools-inspired programs are designed to run processes. Without the -f, you'd have some mighty strange symptoms and a hard time troubleshooting.

Named

named is a DNS lookup engine. This is a tiny bit tougher because /etc/rc.d/rc3.d/S50lrc.local runs a program called startnamed rather than named, and the fact that to run in the foreground, named needs a -f argument. A which command quickly finds /usr/bin/startnamed, which looks like the following:

echo Starting named
named -t /var/named/chroot -u named

OK, the first echo isn't necessary in this "Hello World". Before moving this to daemontools-encore, let's see what the second line, with a -f inserted right before the -t. Run it manually. It doesn't put itself in the background, but instead does nothing until you press Ctlr+C. Good, you'll want the -f. So perform the following steps:

  1. Comment out the startnamed line from /etc/rc.d/rc3.d/S50lrc.local Use four pound signs so that you can tell what you commented out from the original comments.
  2. mkdir /var/service/named
  3. cd /var/service/named
  4. Create and make executable by root the following run file: 
    #!/bin/sh
if netisdown; then
  sleep 10
  exit 1
fi
exec named -f -t /var/named/chroot -u named
  5. chmod u+x run
  6. touch down
  7. Add lk_runsvc /service/named $sleepafter to /root/dtinit.sh, just before the acpidline.
  8. ln -s /var/service/named /service/named
  9. Reboot and check

When you check, be sure to ping 8.8.8.8 to determine connectivity, and then use the lynx browser to browse to different websites. If you can browse to a website by domain name, that's pretty good proof that named is working.

startdbus-daemon

This is /usr/bin/startdbus-daemon, and is a shellscript that does a bunch of simple stuff and then calls dbus-daemon. Here's the original shellscript:

#!/bin/sh

echo Starting dbus-daemon

killall -HUP dbus-daemon >& /dev/null

rm -rf /var/run/dbus
mkdir -p /var/run/dbus

dbus-daemon --system

So we're pretty much going to just copy that into the run script, as follows:

#!/bin/sh

killall -HUP dbus-daemon >& /dev/null

rm -rf /var/run/dbus
mkdir -p /var/run/dbus

exec dbus-daemon --nofork --system

Notice the --nofork in the final command of the preceding run script? That's so it runs in the foreground, like all good daemontools processes should. On another subject, as far as I know, dbus doesn't require the network, and I don't know what process dependencies it might have, so I'm not putting any if statements in the run script. Now follow these steps:

  1. Comment out the startdbus-daemon line from /etc/rc.d/rc3.d/S50lrc.local Use four pound signs so that you can tell what you commented out from the original comments.
  2. mkdir /var/service/dbus
  3. cd /var/service/dbus
  4. Create the run file displayed earlier in this section.
  5. chmod u+x run
  6. touch down
  7. Add lk_runsvc /service/dbus $sleepafter to /root/dtinit.sh, just before the named line.
  8. ln -s /var/service/dbus /service/dbus
  9. Reboot and check

The best way of checking is with ps axjf | less, in CLI mode (before running any X, because a lot of modern window managers or desktop environments run dbus).

sshd

This one's pretty simple, as long as you remember that to run sshd in the foreground, you use the -d option (stands for "debug"). So it's what you've come to expect:

  1. Comment out the /usr/sbin/sshd line from /etc/rc.d/rc3.d/S50lrc.local Use four pound signs so that you can tell what you commented out from the original comments.
  2. mkdir /var/service/sshd
  3. cd /var/service/sshd
  4. Create the following run file: 
    #!/bin/sh
if netisdown; then
  sleep 10
  exit 1
fi
exec /usr/sbin/sshd -d
  5. chmod u+x run
  6. touch down
  7. Add lk_runsvc /service/sshd $sleepafter to /root/dtinit.sh, just before the dbus line.
  8. ln -s /var/service/sshd /service/sshd
  9. Reboot and check

The Remainder

The remainder of /etc/rc.d/rc3.d/S50lrc.local is just run-once type stuff that belongs in a "rc" type script. It's fairly early stuff too.

So we're going to put it all in a run-once service called "bringup". Like all run-once services in this document, we'll terminate it with lk_forever, although, as mentioned many times, with something as sophisticated as daemontools-encore there are better ways of doing it.

This "bringup" service will be added to, more and more, until it replaces the early calls in /etc/rc.d/rc3.d/. Once that's done, the main remaining need for sysvinit is for accurate, non-destructive shutdowns.

But for now, we're just moving the remainder of /etc/rc.d/rc3.d/S50lrc.local into the "bringup" service. Perform the following steps:

  1. Put the command exit 0 immediately after the #!/bin/bash line at the top of /etc/rc.d/rc3.d/S50lrc.local. This effectively comments out the entire file.
  2. mkdir /var/service/bringup
  3. cd /var/service/bringup
  4. Create and make executable by root the following run file: 
    #!/bin/sh

PATH=$PATH:/opt/bin:/opt/sbin

mkdir -p /dev/pts
mount -t devpts devpts /dev/pts

mkdir -p /dev/shm
mount -t tmpfs tmpfs /dev/shm

ifconfig eth0 up
dhclient eth0

lk_forever 3600
  5. chmod u+x run
  6. touch down
  7. Add lk_runsvc /service/bringup $sleepafter to /root/dtinit.sh, just before the sshd line.
  8. ln -s /var/service/bringup /service/bringup
  9. Reboot and check

I have some misgivings about the path here. I think the path modifier statement might need to be moved back to the /root/dtinit script. We'll see.

Whoops! Dhclient

In the preceding section, I ran dhclient inline with a script. That kinda-sorta works, but dhclient is really a daemon that should be managed with daemontools-encore. So delete it from the /service/bringup/run script and give it its own service:

  1. Comment out the dhclient line from /service/bringup/run Use four pound signs so that you can tell what you commented out from the original comments.
  2. mkdir /var/service/dhclient
  3. cd /var/service/dhclient
  4. Create and make executable by root the following run file: 
    #!/bin/sh
exec dhclient -d -q eth0 > /dev/null
  5. chmod u+x run
  6. touch down
  7. Add lk_runsvc /service/dhclient $sleepafter to /root/dtinit.sh, just after the bringup line.
  8. ln -s /var/service/dhclient /service/dhclient
  9. Reboot and check

The Elephant In the Room: rc.shutdown

All this time we've been moving functionality from sysvinit to daemontools-encore, and we've done it quite well. When it's all moved over, theoretically we can copy the content of /usr/local/bin/rc.littkit to /bin/rc.init, boot into Suckless Init, and bang, we're initting via Suckless Init. There are many small problems with that view of the world, and one huge problem: What to put in /bin/rc.shutdown, the program used to shut down the computer, either by reboot or poweroff. Write that script wrong, and you can corrupt your hard disk. Probably nothing you can't journal or fsck your way out of, but not pleasant.

Rather than go into a huge discussion, let me give you the code I used for /bin/rc.shutdown. But before I do, there's one gotcha you should know about: This script depends on killall5, which is a sysv component. If you don't have killall5, you're probably going to need to modify Suckless Init to run /bin/rc.shutdown as part of PID1, instead of forking it. I'll discuss that in other documents. Meanwhile, if you're following along with Plop Linux, you do have killall5, so the following is a reasonable /bin/rc.shutdown:

#!/bin/sh

killall515delay=4
killall59delay=4

waitsecs(){
  numsecs=$1
  echo -n "Wait $numsecs seconds please "
  while test $numsecs -gt 0; do
    sleep 1
    echo -n "."
    let numsecs=$numsecs-1
  done
  echo
}



shutdown_mode=$1  ## CAPTURE "poweroff" or "reboot"

if test $# -lt 1; then
   shutdown_mode="poweroff"
fi


echo "Killing non-tty daemontools-encore services..."
lk_killsvc /service/cupsd 0
lk_killsvc /service/alsactl_r 0
lk_killsvc /service/acpid 0
lk_killsvc /service/named 0
lk_killsvc /service/dbus 0
lk_killsvc /service/sshd 0
lk_killsvc /service/dhclient 0
lk_killsvc /service/bringup 0


echo "Unmounting /dev/pts and /dev/shm..."
umount /dev/pts
umount /dev/shm


echo "Downing the network..."
ip link set dev lo down
ip link set dev eth0 down



echo "Killing klogd and syslogd..."
killall klogd
killall syslogd


echo "killall5 -15 (requesting)..."
killall5 -15
waitsecs $killall515delay


echo "killall5 -9 (murdering)..."
killall5 -9
waitsecs $killall59delay


echo "swapoff -a..."
swapoff -a

echo "remounting read-only /dev/hda1..."
mount -o remount,ro /dev/hda1

if test "$shutdown_mode" = "reboot"; then
  echo "rebooting..."
  /sbin/reboot
else
  echo "powering off..."
  /sbin/poweroff
fi

Once you can shut down with the preceding rc.shutdown, without having all sorts of partition journal messages when you boot up again, you're ready to install and try out Suckless Init.

Install Suckless Init

We've done everything we can without installing Suckless Init. Like all Suckless Tools, Suckless Init is trivial to install if you accept the defaults, which is what I did. To install Suckless Init, perform the following steps:

  1. Browse to http://git.suckless.org/sinit
  2. Download the latest tarball. At the time of this writing, the latest was sinit-0.9.2.tar.gz From this point forward, the path and filename of the tarball you downloaded will be referred to as <tarballpath>.
  3. Be sure you're logged in as root
  4. cd
  5. tar xzvf <tarballpath>
  6. The preceding command created a directory tree for building Suckless Init. That directory tree's filename is referred to as <builddir> for the rest of this section. For instance, my <builddir> is sinit-0.9.2
  7. cd <builddir>
  8. make
  9. If the preceding command gives no errors, then make install
  10. Look for /usr/local/bin/sinit if the preceding command gives no errors. Make sure it was made a when you did your make install.
  11. cp -p /usr/local/bin/sinit /sbin/sinit
    • This command is a convenience factor for later copying.

Suckless Init Architecture

The key distinction of Suckless Init is that it was, and I quote the README file, "sinit is a simple init. It was initially based on Rich Felker's minimal init[1]."

Now, looking over at the blog entry in which Rich Felker exposed his init and gave it a free software license. Rich spends a great deal of time on the subject of "get everything unnecessary for PID1 out of PID1." His 16 line init program does no process management at all. He makes it clear that can be done elsewhere, by another program that isn't PID1 and doesn't have PID1's special privileges and responsibilities. Smarter people than I argue both sides of this "sparse PID1" principle, but until someone can convince me otherwise, I'm a big believer in the sparse PID1 principle.

Now comes Dimitris Papastamos, the author of Suckless Init, who obviously also believes in the "sparse PID1" principle, because his 89 line init basically forks off /bin/rc.init, then spends the rest of the OS' uptime handling signals. Process management is done by /bin/rc.init or something it runs or forks. Sparse PID1. In the case of the document you're reading, /bin/rc.init runs lk_prepare and then execs daemontools.

A special thanks to Rich Felker and Dimitris Papastamos, who both stood behind their belief in a sparse PID1.

Anyway, you've already compiled Suckless Init. Spend a few minutes looking at config.h and sinit.c. This section is a very basic overview of what sinit.c does. Starting at the highest level, here's what sinit.c does:

  1. cd / for safety and minimum surprises.
  2. Process itself ignores all processes: sigfillset and sigprocmask
  3. Forks off /bin/rc.init (spawn(rcinitcmd))
  4. Goes into an infinite loop that inspects signals it receives

Each iteration of the infinite signal inspection loop contains the following steps:

  1. sigwait(&set, &sig) to acquire the next signal that comes in.
  2. If the signal is one of SIGUSR1,SIGCHLD or SIGINT, (for (i = 0; i < LEN(sigmap); i++)) do the following:
    • Execute the proper function for the signal (for (i = 0; i < LEN(sigmap); i++) { if (sigmap[i].sig == sig) { sigmap[i].handler(); break; } )
    • As configured from the factory, the preceding means run /bin/rc.shutdown reboot on receipt of a SIGINT, run /bin/rc.shutdown poweroff on receipt of a SIGUSR1, and a function to reap all zombie children of PID1 using waitpid(-1, NULL, WNOHANG).

To put it in a sentence, sinit.c operates as PID1, and stops all interrupts from interrupting it, forks off /bin/rc.init, and then spins forever, handling any signals it receives.

One more sentence explains the end of the infinite loop: Signals SIGUSR1 and SIGINT cause sinit.c to fork /bin/rc.shutdown, which kills all processes including PID1, thus shutting down the computer.

Here's the bottom line: You just saw a perfectly functional PID1 written in 89 lines of C code. Try not to laugh when, during systems programming bull sessions, someone confidently and authoritatively assures you that PID1 is necessarily tremendously complicated. Just silently smile as you remember hooking up this 89 line C PID1 to daemontools-encore and LittKit, to make a perfectly functional init system.

Suckless Init Hello World

Up to this moment, we've transferred services and commands from sysvinit to daemontools-encore, starting with the latest and going through the earliest, at least the earliest in /etc/rc.d/rc3.d/. It looks like we've relieved sysvinit of all process management duties (we haven't, but it looks that way), so the remaining challenge is to boot to Suckless Init, and then call daemontools-encore.

Learn from my misfortunes --- things go wrong. Therefore, your first use of Suckless Init should do nothing but bring you to a bash prompt so that you can look around and experiment. If you were to try init right into lk_prepare and svscanboot, I think it highly likely that some little thing you forgot would cause you to boot to a state from which you have no input method.

To boot to a bash-only state, perform the following steps:

  1. Shut down your VM
  2. Switch your boot order so that it boots to a live distro (probably Plop-live)
  3. Start the VM
  4. mount /dev/hda1 /mnt
  5. Edit /mnt/bin/rc.init to look as follows: 
      #!/bin/sh
echo;echo;echo;echo;echo
echo PRESS ENTER TO GET COMMAND PROMPT!!!!!!
echo;echo;echo;echo;echo
/bin/bash
exit 0
  6. cd /mnt/sbin
  7. cp -p init init.sysvinit
  8. cp -p sinit init
  9. cd /
  10. umount /mnt
  11. poweroff
      Warning! poweroff works correctly only in sysvinit. Don't call it directly once you're initting with Suckless Init. But as you remember, you're booted to the live CD, and the Plop live CD inits with sysvinit.
  12. Change the boot order to boot from the hard disk.
  13. Start the VM

If you've installed Suckless Init correctly, booting the hard disk tells you to press the Enter key to get a bash prompt, and when you do, you get a bash prompt, overseeing a very defective Linux:

Look at the bright side. You just initted with Suckless Init. That's huge. All you need to do is solve the problems listed above, then run lk_prepare and then svscanboot. It's not all that hard.

The big problems preventing you from running lk_prepare and then svscanboot are an inadequate $PATH and a read-only filesystem. Before remounting the filesystem read-write, it's best to take care of /proc and /sys. And you might as well turn on swap right after /proc and /sys are taken care of.

So, from your defective command prompt, perform the following steps manually:

  1. Add to the path for missing directories. For instance, if /usr/local/bin is missing from your $PATH, perform the following command:
    • export PATH=$PATH:/usr/local/bin
  2. mount -n /proc
  3. mount -n /sys
  4. swapon -a
  5. mount -n -o remount,rw /
  6. mount -f /
  7. mount -f /proc
  8. mount -f /sys

Note:

If you wondered what hat I pulled the preceding steps out of, what I did was look in all the startup files under directory /etc/rc.d/rcsysinit.d. I found that directory from tracing everything. Notice the the preceding steps implement only the most vital elements of the startup code under /etc/rc.d/rcsysinit.d. One thing left out is the facility for periodic fsck runs. For the time, every once in a while boot to a live CD and fsck /dev/hda1, umounted.

Now look around. Notice you can create files on the root tree. The mount command with no arguments gives you a list of mounts. You have a working /proc and /sys directory. Look around. Get used to this environment.

Once you've explored, perform the following two commands:

  1. lk_prepare
  2. /sbin/svscanboot

With the preceding two commands performed, you should have a pretty much functional Linux box, complete with networking, an SSH server, and all the usual suspects. This is because you just ran daemontools-encore, with your ordered startup, and in doing so you ran everything that you transferred from sysvinit to daemontools-encore.

Which leaves only one remaining question: How do you shut this thing down, now that you don't have sysvinit anymore? Here are the two ways:

Danger!

Do not attempt to Ctrl+Alt+Del your VM without first performing the following command:

echo 0 > /proc/sys/kernel/ctrl-alt-del

0 in that file indicates Ctrl+Alt+Del gets delivered as a SIGINT to PID1, which then performs rc.shutdown. If that file doesn't contain 0, then Ctrl+Alt+Del just reboots the computer, without closing and unmounting and the like. You can lose data. Plop Linux puts a 1 in this file upon every bootup.

Note:

Remember that when you're working inside a VirtualBox VM, the way to perform a Ctrl+Alt+Del is to perform HostKey+Del, where HostKey is the key to escape the VM's focus. It defaults to RightCtrl. But reading the preceding danger warning before performing any kind of Ctrl+Alt+Del.

Now that you can boot up to a bash session and issue the proper commands to fully instantiate all services via daemontools-encore, the only thing left to do is put those proper commands into /bin/rc.init, and comment out the call to /bin/bash, the call to exit 0, and the echo commands about pressing Enter. Once you've done these tasks, your rc.init should look something like this:

#!/bin/sh
export PATH=$PATH:/usr/local/bin
#echo;echo;echo;echo;echo
#echo PRESS CTRL+F2 TO LOG IN!!!!!!!
#echo;echo;echo;echo;echo
#/bin/bash
#exit 0



### FROM S00mountkernfs
mount -n /proc
mount -n /sys

### from s20swap
swapon -a


### from S40mountfs
mount -n -o remount,rw / &>/dev/null

# Remove fsck-related file system watermarks.
rm -f /fastboot /forcefsck

#Recording existing mounts in /etc/mtab...
mount -f /
mount -f /proc
mount -f /sys


#Mount everything not depending on thenetwork
mount -a -O no_netdev &>/dev/null


###### TELL USER TO GO TO A VIRTUAL TERMINAL
echo;echo;echo;echo;echo
echo PRESS CTRL+F2 TO LOG IN!!!!!!!
echo;echo;echo;echo;echo


###### RUN DAEMONTOOLS WITH LITTKIT
lk_prepare /service
/usr/local/bin/svscanboot

The Final Result

This section contains the final result of the conversion of Plop Linux from sysvinit to Suckless Init plus daemontools-encore plus LittKit:

rc.shutdown

Upon receiving a SIGINT or SIGUSR1, Suckless Init forks /sbin/rc.shutdown with an argument of either "reboot" or "poweroff". Here's what /sbin/rc.shutdown looks like:

#!/bin/sh

killall515delay=4
killall59delay=4

waitsecs(){
  numsecs=$1
  echo -n "Wait $numsecs seconds please "
  while test $numsecs -gt 0; do
    sleep 1
    echo -n "."
    let numsecs=$numsecs-1
  done
  echo
}


shutdown_mode=$1  ## CAPTURE "poweroff" or "reboot"

if test $# -lt 1; then
   shutdown_mode="poweroff"
fi


echo "Killing daemontools-encore services..."
lk_killsvc /service/cupsd 0
lk_killsvc /service/alsactl_r 0
lk_killsvc /service/acpid 0
lk_killsvc /service/named 0
lk_killsvc /service/dbus 0
lk_killsvc /service/sshd 0
lk_killsvc /service/dhclient 0
lk_killsvc /service/bringup 0


echo "Unmounting /dev/pts and /dev/shm..."
umount /dev/pts
umount /dev/shm


echo "Downing the network..."
ip link set dev lo down
ip link set dev eth0 down


echo "Killing klogd and syslogd..."
killall klogd
killall syslogd


echo "killall5 -15 (requesting)..."
killall5 -15
waitsecs $killall515delay


echo "killall5 -9 (murdering)..."
killall5 -9
waitsecs $killall59delay


echo "swapoff -a"
swapoff -a

echo "remounting read-only /dev/hda1..."
mount -o remount,ro /dev/hda1


if test "$shutdown_mode" = "reboot"; then
  echo "rebooting..."
  /sbin/reboot
else
  echo "powering off..."
  /sbin/poweroff
fi

rc.init

In its default configuration, upon startup Suckless Init forks off /bin/rc.init. Here's what /bin/rc.init looks like:

#!/bin/sh
export PATH=$PATH:/usr/local/bin
#echo;echo;echo;echo;echo
#echo PRESS CTRL+F2 TO LOG IN!!!!!!!
#echo;echo;echo;echo;echo
#/bin/bash
#exit 0



### FROM S00mountkernfs
mount -n /proc
mount -n /sys

### from s20swap
swapon -a


### from S40mountfs
mount -n -o remount,rw / &>/dev/null

# Remove fsck-related file system watermarks.
rm -f /fastboot /forcefsck

#Recording existing mounts in /etc/mtab...
mount -f /
mount -f /proc
mount -f /sys


#Mount everything not depending on thenetwork
mount -a -O no_netdev &>/dev/null


###### TELL USER TO GO TO A VIRTUAL TERMINAL
echo;echo;echo;echo;echo
echo PRESS CTRL+F2 TO LOG IN!!!!!!!
echo;echo;echo;echo;echo


###### RUN DAEMONTOOLS WITH LITTKIT
lk_prepare /service
/usr/local/bin/svscanboot

The Checker Scripts

This setup uses two scripts, in directory /usr/local/bin, to check network status:

netisdown follows:

#!/bin/sh
ping -W 1 -w 1 -c1 10.0.2.2 2> /dev/null 1> /dev/null
rtrn=$?

if test "$rtrn" = "0"; then
  rtrn=1
elif test "$rtrn" = "1"; then
  rtrn=0
else
  rtrn=9
fi
exit $rtrn

And the following is dhcpisready

#!/bin/sh

### KLUDGE, IFCONFIG IS DEPRECATED.

if ifconfig eth0 | grep "\s*inet addr:" > /dev/null; then
 exit 0
else
 exit 1
fi

Both of the preceding scripts are typically used in if statements and loops within other shellscripts, for timing and process dependency.

The dtinit Service

The final two tasks of /bin/rc.init are to run lk_prepare /service and to run /usr/local/bin/svscanboot, which starts up daemontools. Because it's the only subdirectory under /service with a nodown file, the dtinit service runs first. Daemontools runs it first. Here's the /service/dtinit/run script's code:

#!/bin/sh
exec /root/dtinit.sh

That's right. The dtinit service does nothing but run /root/dtinit.sh in the dtinit service's own process with the dtinit service's own PID. Obviously, the contents of /root/dtinit.sh should be copied to the dtinit service, after which the dtinit service should not call /root/dtinit.sh. The reason I've left /root/dtinit.sh separate is in case I need to swap sysvinit back in.

dtinit.sh

/root/dtinit.sh gets executed by the dtinit service. /root/dtinit.sh starts other daemontools services, one at a time, in order, and can also have delays to make sure things are up before continuing. The code for /root/dtinit.sh follows:

#!/bin/sh
initialsleep=0
sleepafter=0

lk_runsvc /service/tty2          $sleepafter
sleep $initialsleep
lk_runsvc /service/syslogd       $sleepafter
lk_runsvc /service/klogd         $sleepafter
lk_runsvc /service/tty3          $sleepafter
lk_runsvc /service/tty4          $sleepafter
lk_runsvc /service/tty5          $sleepafter
lk_runsvc /service/tty6          $sleepafter
lk_runsvc /service/bringup       $sleepafter
lk_runsvc /service/dhclient      $sleepafter


while ! /usr/local/bin/dhcpisready; do
   sleep 1
   echo waiting for network to come up
done


lk_runsvc /service/sshd          $sleepafter
lk_runsvc /service/dbus          $sleepafter
lk_runsvc /service/named         $sleepafter
lk_runsvc /service/acpid         $sleepafter
lk_runsvc /service/alsactl_r     $sleepafter
lk_runsvc /service/cupsd         $sleepafter

#lk_runsvc /service/tty1          $sleepafter

lk_runsvcs /service              $sleepafter

lk_forever 3600

The preceding brings up an early tty, tty2. For practical purposes, the other tty's come about the same time, but they don't need to. They could be much later, and as long as you have one early tty, you can access the computer to fix it.

The commented out reference to tty1 is for ease in troubleshooting through repeated reboots. In such a case, you'd uncomment the reference in /boot/dtinit.sh and you'd make a tty1 service that uses the -a root argument for agetty in order that you don't need to repeatedly log in. Once you're done with troubleshooting, you can create file /service/tty1/reallydown file so that it doesn't come up, and re-comment it in /boot/dtinit.sh.

The TTYs

The TTYs are virtual terminals, the command line interfaces you get to with Ctrl+Alt+F2 through Ctrl+Alt+F6. They're created by respawning the agetty command. The code for /service/tty6/run follows:

#!/bin/sh
exec /sbin/agetty tty6 9600

You can clone the preceding to /service/tty2 through /service/tty5, obviously changing the first argument of the /sbin/agetty command accordingly.

Reminder:

You're not really creating /service/tty6 or any of the others. You're creating /var/service/tty6, and then making /service/tty6 a sylink to /var/service/tty6. Remember also that the constructed services don't need to be under /var/service, they could be under anything that's on the root partition and is readable and writeable.

Note:

The reason I included the 9600 baud rate in all the ttys is because that's how they were in the original /etc/inittab. The baud rates really aren't necessary in virtual terminals, and once you get everything running well you can simply delete the baud rates from the tty run commands.

As discussed before, you might want to make a tty1 for the purpose of instant login for troubleshooting through repeated reboots. If you do, it would look like this:

#!/bin/sh
echo;echo;
exec /sbin/agetty -noclear --noissue -a root tty1 9600

For security reasons, obviously you need to disable this no-login terminal once you're done troubleshooting.

The syslogd Service

#!/bin/sh
exec syslogd -n -m 0

The klogd Service

#!/bin/sh
exec /sys/klogd -n

The bringup Service

#!/bin/sh

PATH=$PATH:/opt/bin:/opt/sbin

mkdir -p /dev/pts
mount -t devpts devpts /dev/pts

mkdir -p /dev/shm
mount -t tmpfs tmpfs /dev/shm


##### BEGIN NETWORK SETUP
hostname -F /etc/hostname
ip link set dev lo up
ip link set dev eth0 up
#####  END  NETWORK SETUP

lk_forever 3600

The dhclient Service

#!/bin/sh
exec dhclient -d -q eth0 > /dev/null

The sshd Service

#!/bin/sh
if netisdown; then
  sleep 10
  exit 1
fi
exec /usr/sbin/sshd -d -q

The dbus Service

#!/bin/sh

killall -HUP dbus-daemon >& /dev/null

rm -rf /var/run/dbus
mkdir -p /var/run/dbus

exec dbus-daemon --nofork --system

The named Service

#!/bin/sh
if netisdown; then
  sleep 10
  exit 1
fi
exec named -f -t /var/named/chroot -u named

The acpid Service

#!/bin/sh
exec /usr/sbin/acpid -f

The alsactl_r Service

#!/bin/sh
/opt/sbin/alsactl restore
lk_forever 3600

The cupsd Service

#!/bin/sh

# NO USE RUNNING CUPS UNTIL THE NETWORK IS FUNCTIONAL
if netisdown; then
	# NETWORK NOT UP, WAIT AWHILE AND EXIT.
	# svscan will try again after run script exits.
	# 10 seconds is a good interval to wait
	# for the network to come up.
	sleep 10
	exit 1
fi

exec /opt/sbin/cupsd -f -c /opt/etc/cups/cupsd.conf

Yes, It Works With The Rich Felker Init

Curiosity killed the kitty, and I just had to find out whether I could swap Rich Felker's 16 line init program instead of Suckless Init. First step, read the source code. Rich Felker's appears more clever, Suckless Init appears more like the straightforward code I write, but a few minutes of examination indicated to me that the main functional difference is that Suckless Init responds to incoming signals, whereas Rich Felker's Init just ignores them. Which also means that Rich Felker's Init needn't have code to deal with signals, like invoking /bin/shutdown or running the reap() function. With Rich Felker's Init, you need to run /bin/rc.shutdown manually.

Rich Felker's Init's failure to respond to kill commands gives us a method of determining which init we're working with. Cool!

So what I did was this: I compiled Rich Felker's code:

gcc -o felker.bin -Wall felker.c

I kept running the preceding until I got no errors, no warnings. I had to add #include<wait.h> in order to kill the warning.

Next, I copied felker.bin to /f. This makes explicit booting from the LILO prompt easier.

Then, because felker.c forks off /etc/rc, I created and made executable the following /etc/rc:

#!/bin/sh
exec /bin/rc.init

The preceding means that running /etc/rc is, for all practical purposes, just like running /bin/rc.init. The reason I didn't simply change Felker's code to call /bin/rc.init directly is I suspected I might need some diagnostic commands specifically when running Rich Felker's Init.

So anyway, I rebooted, and at the LILO prompt I typed:

ploplinux init=/f

It booted up exactly as with Suckless Init, but when I used kill to send PID1 a SIGINT or SIGUSR1, instead of rebooting or powering off like Suckless Init would, it did nothing at all. This proves I was initted to Rich Felker's Init. I powered it off by typing the following command as root at the command prompt:

rc.shutdown poweroff

Rich Felker's Init was a success on my Plop system. Which brings up the question: "Should I init with Rich Felker's Init, or with Suckless Init?" The tale of the tape shows felker.c with 16 non-blank lines of code, and sinit.c with 76 non-blank lines of code. One way of looking at it is that felker.c has 450% more code. Another way of looking at it is they're both tiny and both very understandable. To me, length isn't an issue in either one. Suckless Init gives me signal handling, and even gives me the opportunity to have PID1 react to additional signals. I think that would be unwise of me, but I could do it.

So for demonstration of a really tiny init, I'd use Rich Felker's Init. For something I'd actually use, assuming there was some reason I wouldn't use runit, s6 or Epoch, I'd use Suckless Init.

Congratulations

Congratulations! You've 100% replaced your old sysvinit init system with a combination of the 89 line Suckless Init, daemontools-encore, and LittKit. You now have an instinctual feel for what PID1 does, and what process management does, and how the first passes off to the second. You've seen first hand how PID1 is responsible to hang around forever, listening to signals, and when it gets the appropriate signal, it must shut down the computer: In this case by forking off /bin/rc.shutdown with an argument of either poweroff or reboot

You feel at home in those init discussions where they argue whether process management should be done by PID1, or by a forked process (in this case /bin/rc.init calling daemontools-encore).

In a "can't boot" situation, you understand how to replace the current init with /bin/bash to see whether boot even got to the hard disk init, or whether it failed in the bootmanager or the initramfs. This gives you troubleshooting tools others just don't have. If need be, you can install Suckless Init and a proper /bin/rc.shutdown, and use that environment to further investigate problems happening at or after the original disk init.

If you ever decide that the init system your distro gives you doesn't fit your needs, what you did in this document gives you the confidence and knowledge to install alternative init systems, especially daemontools-inspired init programs runit and s6.

Congratulations. Seriously. I'll bet you less than 1/10 of those who started reading this doc got past the third section. I'll bet you that of those who got past the third section, 19 out of 20 gave up before the end, because they considered it too tedious. That leaves you, the 1 out of 200, the elite 0.5%, DIY enough to complete this, and really understand the init part of what's under the Linux hood. In a world where people think they're "technical" because they've learned the cockpit controls of the latest technology, you know you're the real deal because you can operate under the hood.

[ Training | Troubleshooters.Com | Email Steve Litt ]