Troubleshooters.Com®, Linux Library
and Init System Choices Present:

The Manjaro Experiments

Copyright © 2014 by Steve Litt

CONTENTS:

December 17, 2014

As I begin this document, it's December 17, 2014. Here's my current situation:

My Manjaro 8.11 64bit OpenRC edition experimental computer boots with a combination runit/OpenRC init system. It boots Grub to command prompt in 10 seconds, and shuts down LXDE to poweroff in 5 seconds. This is a circa 2008 commodity box, with an AMD Athlon II X2 250 Processor. According to cpubenchmark.net, this processor has about one third the computing power of a current Intel I5. This computer has 4GB of RAM, and an ordinary 3.5" hard drive. Equipped with its runit->OpenRC init and its dbus-less LXDE, it's a reasonable desktop computer with sound, video, Internet.

[root@openrc ~]# ps ax | grep -v grep | grep dbus
[root@openrc ~]#

PID 1 is runit. Runit has three "stages", each represented by a shellscript. Stage 1 does one-time bootup tasks. Stage 3 does shutdown tasks. Stage 2 manages processes, and is a heck of a lot like daemontools. My stage 1 script, /etc/runit/1, follows:

#!/bin/sh
# system one time tasks

PATH=/command:/sbin:/bin:/usr/sbin:/usr/bin

rm /tmp/sentinels/*

#/usr/bin/openrc sysinit
#/usr/bin/openrc boot
#/usr/bin/openrc default

### START devfs AND sysfs, WRITE SENTINEL FILES
/usr/bin/openrc minimum
rm /tmp/sentinels/*
touch /tmp/sentinels/devfs.sent
touch /tmp/sentinels/sysfs.sent

#runsv /service/udevd

### SET UP NETWORK WITHOUT systemd, NetworkManager, dbus or wicd ###
/root/upnet.sh

/etc/init.d/rcS
/etc/init.d/rmnologin

touch /etc/runit/stopit
chmod 0 /etc/runit/stopit

The preceding is pretty simple, all things considered. The stuff about /tmp/sentinels is my attempt to record which tasks have already been done, because, as far as I know, runit has nothing like OpenRC's "provides" keyword, so I actually need to leave something that can be tested.

The three commented openrc calls (sysinit, boot and default), would perform almost all initialization using OpenRC, if they were uncommented. In OpenRC, sysinit, boot and default are called "runlevels", which is a different definition than used with the more familiar sysvinit init system, where you tend to run one specific runlevel, instead of consecutively running three.

The point is this: I wanted to transfer as much responsibility as possible from OpenRC to runit, so I made a new OpenRC runlevel, called minimum, that starts only these three things:

  1. devfs
  2. sysfs
  3. udevd

With just those three underpinnings accomplished, I can do almost everything else with runit.

It would have been easy enough to run NetworkManager in order to set up networking, but NetworkManager depends on dbus, and I'd prefer to minimize my reliance on dbus. As a matter of fact, dbus doesn't run on my current Manjaro box, although that might change later. So, to start up networking, after instantiation of devfs, sysfs, and udevd, the runit stage 1 script runs a shellscript called /root/upnet.sh, which looks like this:

#!/bin/sh
hostname -F /etc/hostname
ifconfig lo up
ifconfig eth1 add inet 192.168.100.88 broadcast 192.168.100.255 netmask 255.255.255.0
ip link set dev eth1 up
route add default gw 192.168.100.96

NOTE:

The preceding script uses the ifconfig command, which is now deprecated in favor of the ip command. I've made a better Network Start Script that uses only ip commands.

A couple notes about the preceding. First, the fact that this shellscript is in /root is just an experimental convenience. It can be anywhere. Second, this script hardcodes all the information. It would be easy enough to use dhclient, to connect the wired ethernet port via DHCP. Third, I've already created a script to connect to a specific Wifi ssid via wpa_supplicant, using DHCP, no NetworkManager or wicd (both of which now depend on dbus) required. For use on the road, it wouldn't be particularly hard for me to create a small front end to wpa_supplicant and iwlist to pick from a list of available Wifi networks and log into one. No NetworkManager or wicd required, and no consolekit (or whatever they're calling it these days), because I can manage that via groups and sudo.

This Document's Purpose

The purpose of this document is so others can follow in my footsteps, but take minutes or hours to accomplish what it took me days or weeks to do with research and experimentation. Therefore, this document's content must be explicit and unambiguous. If you run into any ambiguities in this doc, or if following its suggestions doesn't work for you, please email me so I can make it clearer. I have a dog in this fight: I want to make sure that anyone wanting to choose their own init system can do so, preferably on any distro they choose.

This doc will be an ongoing project, as I learn more and more about init systems via my Manjaro experimental computer.

Haters, Whingers, Demotivators and Trolls

If you say "I don't want systemd on my computer", the insults fly quick and fast. I don't just mean on the mailing list of a distro that's gone systemd, such as Debian. I mean on LUG lists and IRC channels, and even on lists and channels serving those wanting a systemd alternative. "Hater." "Whinger." "Demotivator." "Troll." "Offtopic." "Take it to the offtopic list." "Spammer." These characterizations are unavoidable to the person who chooses to kick systemd off their computer.

So let me state my position: I'm the one who will choose the init system for my computer!

Hey, I live in a free country. I use free software. I'm free to modify my software to fit my needs. Tell me one more time just so I understand: Why does wanting choice, including the choice to kick systemd off one's computer, make one a hater or a whinger?

You know, the Debian init system wars were never about systemd. They were about choice. When systemd started subsuming more and more parts of the operating system, our choices were foreclosed, or at least the choices that were easily done with the package manager. The fight was about that foreclosure.

So the next time you hear people characterizing alternative-init fans as "haters" and "whingers", the question you might want to ask the epithet hurler is this: "What do you have against choice?"

And understand that as you read this document, it's about choice.

Init System Primer

The Init System is the program run directly by the kernel. It's always PID (Process IDentifier) number 1. After Grub or LILO or whatever hands control to the kernel, the kernel initializes itself, runs any necessary kernel processes (the processes the ps command prints in square brackets), the kernel runs exactly one program, and then just hangs around waiting for requests. That one program is the Init System. The Init System first runs a bunch of setup (as in runit stage 1), then runs and manages a bunch of processes (as in runit stage 2), all the while looking out for interrupts like Ctrl+Alt+Del, or programmatic shutdown/reboot commands like halt, reboot, /sbin/shutdown -h now, init 0, init 6, openrc shutdown, runit-init 0, runit-init 6, and the like. When any of those interrupts or commands are heard, runit stage 3 is run. Runit stage 3 shuts down or reboots the machine in an orderly fashion.

Runit stage 2 also manages processes that it started. This means that it handles logging of those processes, restarts them if they crash, and gives you tools to start, stop, and restart individual processes, as well as temporarily disabling processes, even through reboots, without disrupting your configuration.

And, perhaps best of all, runit manages all these processes without need for the user or admin to worry about PIDs. If runit ran it, runit knows how to find it, without a burdening the user or admin with the need to track PIDs.

The init program filename can be anything. Traditionally, it's been called /sbin/init, but it can be called anything. Simply append the following to your Grub boot line:

init=/path/to/program/myinit_program
.

Or you can back up your existing /sbin/init, and then copy /path/to/program/myinit_program to /sbin/init.

A few pieces of information: The systemd init program is tucked several directories deep, so if you're switching over to a different init system, once everything's debugged with that system, you're probably better off changing the init program location permanently within Grub or LILO. Also, it might seem a good idea to have /sbin/init be a symlink to the real init program. That can fail. Specifically, it fails with runit, where a symlink brings up a jailed busybox emergency environment.

In summary, when the kernel is done initializing itself, it runs exactly one program, the init program. The init program sets up various necessities, and then runs and manages all necessary processes. If hit with specific interrupts or commands, it cleanly shuts down or reboots the computer.

Make a Damage Control Plan

Step 2 of the Universal Troubleshooting Process is Make a damage control plan. The purpose is to be able to back out and/or minimize any damage you cause during your troubleshooting (or experimentation). The Damage Control Plan must in place before work commences. In the case of init system experimentation, here is the Damage Control Plan I use:

Work on an experimental machine

If you never fall off your skateboard, you're not really going for it. In exactly the same way, if you never make your box unbootable during init system experimentation, you're not really going for it. And of course, whenever you bork your bootability, the restoration presents the possibility of inconvenient unscheduled backup at the best, and data loss at the worst.

If you do your experimentation on an experimental machine, the worst that can happen to you is you reinstall the OS and redo the successful steps you've taken. It's worry free.

Doing init experimentation on a virtual machine is just as good, always assuming:

  1. You don't accidentally perform a step on the host OS or the wrong VM.
  2. Your experimentation doesn't include VM hosting.

Have a known good System Rescue CD at the ready

With a known good, tested, bootable CD or DVD with the System Rescue CD system on it, you can bust back into almost any borked OS, as long as the filesystems are still good. So, when you boot to an init that won't even bring up a terminal, no problem: Boot System Rescue CD, mount the partition(s), back out your last change, and reboot to the hard drive. No muss, no fuss, no bother.

Obviously, this is time consuming, involving two boots, so what you might want to do is have a known good init at, let's say, /sbin/init.knowngood, so you can simply tack init=/sbin/init.knowngood on the end of your grub2 boot command. But if all else fails, if you have a CD formatted to System Rescue CD, you'll probably be able to bust back in and fix the problem without needing to reinstall.

Keep track of the steps you've done

You can't back out what you've forgotten, so be sure to record each step you took. And obviously, don't record it on the experimental computer, which could become unbootable. Pencil and paper, or another computer, or a networked drive, are best. And save often. Besides the benefit of being able to back out your last change or two, recording your steps with their results helps you document the correct process later, and helps you quickly reproduce it on another computer.

Make It Easy On Yourself

Experimentation is no place to load yourself down with every difficulty known to man. During early experimentation, you would no more use a multi-partition setup than you would learn to drive with a stick shift. Can it be done? Yes, in both cases. Will it add more than the sum of the challenges in terms of time to mastery? Yes, much more.

Init system experimentation is not the time to gratuitously use non-defaults. Where possible, use the defaults, and if possible use only a single partition, for /, and have your whole OS and /home on that partition. When you need to mount the OS partition from System Rescue CD, you'll be very glad you need to mount only one partition.

Make it easy on yourself. When experimenting, delay the exotic.

Lessons Learned As Of 12/17/2014

Here are some of the lessons I've learned about init systems as of 12/17/2014:

There are more than three init systems.

Most of those involved in the Debian CTTE Deliberations implied that the only init system choices were sysvinit, Upstart and systemd. Nothing could be farther from the truth. Here are the init systems I know about:

  1. runit
  2. OpenRC
  3. sysvinit
  4. Upstart
  5. systemd
  6. S6
  7. perp
  8. nosh
  9. Epoch Init System

If anyone tries to argue with you based on there being three possible init systems, smile and nod and try not to laugh.

All init systems have grossly inadequate documentation.

Systemd has been (rightly) attacked as having inadequate documentation. But so does upstart. And sysvinit, if you need to do anything the slightest bit creative. S6's docs were such that I installed it, but didn't know what to do next, and had to abandon it. I couldn't even get nosh compiled, and I think that's mostly because of bad docs. Runit's documentation is so bad that I had to learn it by experimentation, even though I know its ancestor daemontools inside and out. OpenRC has some good docs, mostly courtesy of Gentoo and Arch, but not really good enough to understand it.

Beneath all of that, very few have explained the high level architecture of an init system. I think Runit comes the closest to a good high level explanation, at least if you combine it with djb's and my docs on daemontools. But it's not good enough to develop a critical mass of Linux user understanding.

This universal lack of documentation produces several problems:

If you know, really know, the nuts and bolts of any init system, please document it. And document it for someone not already familiar with it. Don't say "this function takes four arguments", but instead show the exact syntax and give examples. Too many great init systems have languished because nobody had the time to learn their details by experimentation.

Linux users' ignorance of init systems is astonishing.

I can't really say more than the title of this subsection. Now that I have a sliver of init system knowledge, I appreciate the black box most see it for. And of course there are those who proclaim knowledge they don't have.

I'm ignorant of init systems.

I'm no exception to init system ignorance. I've never coded an init system. I have little knowledge of how the init system handles interrupts and other inputs (from FIFOs and sockets, for instance). My sole claim to fame is that I've booted from sysvinit->OpenRC, runit->OpenRC, and that I've reduced OpenRC to running just three OpenRC scripts (devfs, sysfs and udevd). I'm not an authority, I'm just a guy five feet ahead on the trail, wielding a machete to cut through the brush, so that those who follow can do less machete work.

Much init system complexity is fluff and edge case CYA.

In case you didn't catch it when I said it earlier in this document, my experimental computer, which is a reasonable desktop computer running LXDE, starts these programs or processes:

With just the preceding, I've made a fairly good desktop computer. Is it "ready for prime time?" No. But it wouldn't take much to get it there:

A lot of things that were originally intended to be started by OpenRC, but are not started as part of my OpenRC setup, appear to "just work" without running them. Urandom, the clock, the /proc directory, and sound are just a few. Will I later find problems with these things? Maybe, but if so, most of these things are very easy to either run, or make substitutes for.

Which brings up this question: If I got this system working using only 10% of the normal processes, why all these processes? My theory is that all these processes are so that every conceivable use case will work, without programming, scripting, or anything outside the packaging system. Of course, this makes the packaging system so complex that, personally, I'd rather be writing scripts.

The purpose of a lot of this complexity is to make a user interface where all configuration is done with successions of mouse clicks rather than config files or shellscripts. Where the user never needs to become root, because something like consolekit keeps a granular list of what the user can configure. Where separate programs magically react to each other. For the right person, these are all nice benefits. But what's the cost? My experimentation tells me the cost is a tenfold increase in the number of daemons the init system needs to spawn, and therefore a much more complex and fragile software environment.

Is it worth it? Sure, for some. They use Gnome or Unity or Windows or OS-X to drag and drop and click to their heart's content, never dealing with a command prompt or editing a config file. They consider any other way of working unprofessional. These users need lots of daemons, each with a complex init script or unit file to take into account every use case, because the user can't or won't configure or write a simple shellscript.

But for others, starting with a simple system and personally addressing their own individual needs with a shellscript here, a config file there, and maybe a few Python programs, is much easier. And much less brain-taxing. This isn't everybody's cup of tea, but for me, it means a simple system designed specifically for me. And I know a lot of other people who think the same way.

I'd compare the "daemon for every conceivable use case" with prescription medicine. One or two pills a day, fine. Five, well, maybe. But some people take ten pills a day, and the pills interact with each other in strange and undiagnosable ways, and the next thing you know, those pills have brought the patient a whole new set of problems.

Manjaro's Pretty Cool

This lesson learned has its own section

Debian's Seen Better Days

This lesson learned has its own section

Manjaro Experiments, Pre-20141217

In a way, these experiments have gone on for months: Ever since I heard about systemd's architecture and decided that architecture wasn't something I wanted on my computer. My experiments took two paths:

  1. Experiment with various systemd-free distros.
  2. Experiment with replacing systemd myself.

After trying Funtoo, (I'd tried Gentoo years before and didn't like it), Porteous, PCLinuxOS, OpenBSD, FreeBSD, PC-BSD, Arch, and a couple others, I determined that 2015 would be slim pickings for the guy who wanted fast, easy installation and a good package manager, and no systemd. That being said, these experiments proved to me that I could base my business on PC-BSD. This was a relief.

But the systemd wars on the Debian-User mailing list, combined with what I considered the breathtaking stupidity of the arguments I'd heard there, and the April Fools Joke architecture of systemd itself, got me curious enough to find and install alternate init systems, even after I'd decided on PC-BSD. After all, systemd proponents had stated quite firmly that no modern computer could be without the assistance of systemd. That got me curious, and more important, it got me ornary.

I began trying to replace init systems.

One of my first attempts, and this was probably a couple months ago, was to use the code near the bottom of this blog. I kinda-sorta got it to run bash, but that was about it. I lacked the knowledge to go further.

I also tried the yocto init shellscript, but failed to get it to put me into a command prompt. I compiled nosh and S6, but had insufficient documentation to get them to assist in booting the computer. At least without a crazy amount of trial and error experimentation.

The turning point came when a guy on the Freenode #debianfork IRC channel told me that systemd on the Manjaro distro could easily be replaced with OpenRC, and recommended I tune in to the #manjaro-openrc channel. I followed some instructions to replace systemd with OpenRC, with patchy success. Then somebody told me there was now a Manjaro 8.11 OpenRC ISO, so I downloaded and installed it.

NOTE:

Over time, the link for the latest Manjaro OpenRC ISO will change. You can always find links to the latest ISO at https://forum.manjaro.org/index.php?board=50.0, in the "Sticky Topics" section.

Having a working OpenRC init system was experimental heaven. I could now make one change and see its results, instead of having to guess exact syntax and procedures for ambiguous documentation. I became conversant with the rc-update and rc-status commands. I installed a service that started daemontools, and started my Virtual Terminals from daemontools after commenting them out of /etc/inittab. It worked.

One of my first moves with Manjaro's sysvinit->OpenRC setup was to replace sysvinit with a shellscript:

#!/bin/bash

if test "$1" = "0"; then
   /usr/bin/openrc shutdown
   exit 0
elif test "$1" = "6"; then
  /usr/bin/openrc reboot
  exit 0
fi

echo TOP OF INIT.SH
env | grep -v COLOR 
echo 
echo $@ 

env | grep -v COLOR > /etc/initargs.log
echo >> /etc/initargs.log
echo $@ >> /etc/initargs.log
sleep  5

# System initialization, mount local filesystems, etc.
/usr/bin/openrc sysinit

# Further system initialization, brings up the boot runlevel.
/usr/bin/openrc boot

/usr/bin/openrc default

echo IF YOU READ THIS, YOU'VE FALLEN THRU TO THE BOTTOM.

The preceding init script, which obviously contains lots of diagnostic prints and logs, kinda-sorta worked, booted up fine, but had problems on shutdown or reboot. Nonetheless, I'd proven the concept of using a non-sysvinit PID1 with OpenRC, although obviously OpenRC was doing all the work.

But with OpenRC, my init process still seemed like a black box. My ultimate goal was to init the box with a daemontools-like init system. At first I had OpenRC hand over control to daemontools, but that caused a lot of problems.

I managed to get runit to compile based on the install instructions on the runit website. Then I managed to deploy it using the PID 1 replacement instructions. I'll cover details and troubleshooting of this step later in this document. It wasn't easy for me, but now that I've been through it, I can explain how it can be easy for you, later in this document. Anyway, as per the instructions, the only thing runit did was expose Virtual Terminal 5 when I pressed Ctrl+Alt+5.

Actually, it wasn't that easy, because I had a wrong run command in my runit stage 2 getty-5 directory, but I finally troubleshot that.

My next step was to add the following to /etc/runit/1:

/usr/bin/openrc sysinit
/usr/bin/openrc boot
/usr/bin/openrc default

In other words, I was using OpenRC exactly how it was supposed to be used: To do everything except receive the handoff from the kernel. It worked perfectly.

Next, I experimented to see how many OpenRC actions I could remove and still successfully boot, initting from runit with less and less help from OpenRC. It turns out I needed to have OpenRC run and manage udevd, and udevd required sysfs and devfs. So I made a brand new OpenRC run level called "minimum", that contained links for sysfs, devfs and udev, and replaced the three openrc commands in /etc/runit/1 with the following single command:

/usr/bin/openrc minimum

After doing that, the computer booted up to the point where I could use ifconfig and the ip and route commands to set up the network. And of course I added a sshd service to stage 2 of runit. This gave me a desktop computer that, if you didn't delve too deeply into details like the fact that it never prophylactically runs fsck, worked almost identically to my Debian Wheezy daily driver.

Using OpenRC with Manjaro

I imagine that with enough knowledge you can install OpenRC on any Linux distro. I chose Manjaro because it's an excellent desktop distro, very similar to Debian, and it has a ready-made OpenRC edition available at https://forum.manjaro.org/index.php?topic=19215.0. From this point, I'll assume you have OpenRC installed, complete with the init scripts necessary to run your computer.

OpenRC is a cool init system. It has facilities very similar to what I've read systemd has for calculating process dependencies. On Manjaro, all its init scripts are kept in /etc/init.d. These init scripts are quite similar to those for sysvinit, except they use #!/usr/bin/openrc-run instead of #!/bin/sh. OpenRC's runlevels are used differently than those for sysvinit. With OpenRC, each runlevel runs a set of tasks, and in booting up, you'll typically run several runlevels, such as "sysinit", "boot" and "default". When shutting down, you typically run runlevel "reboot" or "shutdown".

These runlevels are implemented similarly to those in sysvinit: Through symlinks. All the init scripts are in /etc/init.d, and the init scripts run by a runlevel, let's call it myrunlevel, would be symlinks in /etc/runlevels/myrunlevel to the files in /etc/init.d. Here's an example for my runlevel called "minimum":

[slitt@openrc ~]$ ls -l /etc/runlevels/minimum
total 0
lrwxrwxrwx 1 root root 17 25.11.2014 14:42 devfs -> /etc/init.d/devfs*
lrwxrwxrwx 1 root root 17 25.11.2014 14:42 sysfs -> /etc/init.d/sysfs*
lrwxrwxrwx 1 root root 16 16.12.2014 21:07 udev -> /etc/init.d/udev*
[slitt@openrc ~]$

You can add or remove processes from levels by deleting or making symlinks. Also, the rc-update command is a more user friendly way of dealing with those symlinks, and is portable in that it works no matter where your distro keeps those symlinks. You can make a new runlevel just by making a new, appropriately named directory, below /etc/runlevels, and creating the desired symlinks inside of it.

Technically, OpenRC isn't a complete init system. It can't receive a handoff from the kernel, because it doesn't do certain things that a PID1 must do. So the way you use OpenRC is to use a "real" init system, such as sysvinit or runit, to make all the necessary OpenRC calls, after which OpenRC manages all the processes. I can't tell you offhand whether systemd can act as the PID1 and hand off to OpenRC because systemd tends to do a lot more than just init, and I don't know how separable the parts are.

Some day I'd like to use the code near the bottom of this blog to init and then hand off to OpenRC, perhaps by making /etc/rc look like this:

#!/bin/sh
/usr/bin/openrc sysinit
/usr/bin/openrc boot
/usr/bin/openrc default

My impression is that if you get OpenRC installed, it's both powerful and trivial to administer if somebody has written the OpenRC init scripts for you. Those init scripts tend to be as complex as those for sysvinit.

When I hear people brag about systemd, those boasts usually involve fast boots, clean process dependency handling, and parallel initialization of processes. OpenRC fulfills all three of those criteria. And then OpenRC gets the hell out of the way.

Daemontools Basics

This section contains some very basic facts about daemontools, a process management software written by Daniel J. Bernstein (djb)

Runit is a daemontools-like init system. So are nosh and S6. If you're truly interested in alternate init systems, it behooves you to learn about daemontools.

In my opinion, the best daemontools documentation is contained djb's daemontools documentation and Steve Litt's daemontools documentation. That being said, there are many other excellent sources.

Daemontools daemonizes and controls foreground processes, not background processes. When starting a normally background process from daemontools, find that process's command line argument to put it in the foreground. For instance, /usr/bin/sshd -d or /usr/bin/crond -n. Daemontools and runit have some kludgy workarounds to foregroundize processes that give no choice but to run in the background, but these kludges might not work, might be unstable, or might produce side effects. When you can, run it in the foreground. When you can't, think long and hard whether you need that daemon.

Adding a daemon (service in daemontools-speak) requires creating a directory tree, and then symlinking that directory tree to a special directory (/service by default). Here's an example:

ln -s /etc/sv/sshd /service/sshd

In the preceding, you did all the daemon configuration in tree /etc/sv/sshd, and then, to make it actually function, you symlinked it to /service/sshd

The default symlink target, /service, can easily be changed. This is important to those who don't like miscelleneous directories created straight from the root.

The tree you create to assemble a daemontools-managed daemon is built like this:

/etc/sv/sshd
|-- log
|   |-- run
|   `-- supervise
|       |-- control
|       |-- lock
|       |-- ok
|       |-- pid
|       |-- stat
|       `-- status
|-- run
`-- supervise
    |-- control
    |-- lock
    |-- ok
    |-- pid
    |-- stat
    `-- status

The supervise directories in both sshd and sshd/log are created and managed by daemontools, after you symlink sshd into the /service directory. You needn't construct or modify the supervise directories, except in very unusual troubleshooting situations when the system gets into a bad state. Your work is to create the run scripts, which are usually pretty simple.

Undocumented Runit Rules of the Road

First, I highly suggest you obtain runit from what Debian would call "the upstream", in other words, runit author Gerrit Pape's website, rather than from a package. When you use Gerrit Pape's code with the instructions on that page, you're dealing with a known entity, and not something a package manager can screw up (or abandon). If you or your employer or customer really have a problem with /service directory being right off the root, this can be changed. But...

I'd highly suggest you follow Gerrit Pape's instructions exactly, even if they produce directory structures that are against your best practices. Once you have everything working, it will be trivial to adjust all directories to conform to your needs and principles.

The next thing to understand is that runit probably won't run the first time. That's OK, you can troubleshoot it.

Another thing: This section assumes you named the service for running tty5 "getty-5", and the directory in which you assemble all services is called /etc/sv, and your service directory, in which you create symlinks to the directories contained in /etc/sv, is called /service. If you diverge from any of these conventions, you'll need to change your diagnostic tests to reflect those divergences.

To maximize the chance of your computer displaying a reasonable interface for troubleshooting, I suggest you enable virtual terminals 2 and 3 within /etc/inittab. This enablement would look something like this:

c2:2345:respawn:/usr/bin/agetty 38400 tty2 linux
c3:2345:respawn:/usr/bin/agetty 38400 tty3 linux

If /etc/inittab enables virtual terminals (tty) 1, 4, 5 and 6, I suggest you comment them out. Commenting out tty1 means the boot display remains visible on Ctrl+Alt+F1. Commenting out tty 4 through 6 gives you room to instantiate those ttys with runit. Be sure to reboot after making those changes to /etc/inittab.

Gerrit Pape's instructions yield a proof of concept, a "Hello World" if you will, that does nothing but instantiate tty5. No network, no other niceties, just tty5. Its a proof of concept. Assuming you commented tty5 out of /etc/inittab, if Ctrl+Alt+F5 brings you to a login prompt, runit is acting partially or completely correctly. If it brings you to an empty blank screen, runit is not working correctly: Time to troubleshoot.

Init systems are complex, and require a reboot for every troubleshooting iteration. Process managers are complex, and require a restart of the whole process manager for every troubleshooting iteration. You need to attack this with a troubleshooting ladder, going from the highest level, cheapest (quickest) diagnostic tests to the lowest level, difficult and time consuming tests:

  1. Note that Gerrit Pape's example tty uses the getty executable, but your distro might not have getty command, but instead might have agetty or mingetty instead. Use the executable you have in order to enable your service run.
  2. For the purposes of this troubleshooting ladder, name your tty5 service "getty-5". Once everything's working, you can name it anything you want.
  3. The first few tests should happen before you create symlink /service/getty-5 for file /etc/sv/getty-5. Also, you might use a different directory instead of /etc/sv/getty-5, and that's perfectly OK: The concept's the same, just change directory names in your diagnostic tests.
  4. Run the exec line of the service's run script, whose filename is probably something like /etc/sv/getty-5/run, at the command prompt, minus the exec part of the command. Run it as root at the command prompt. If this doesn't work, meaning it doesn't produce a login prompt when you press Ctrl+Alt+F5, troubleshoot until it does. Use any error messages the command produces, as well as various ps commands, to narrow it down. 
    • [root@openrc ~]# /sbin/agetty 38400 tty5 linux
    • The preceding command should run without returning, and should throw up a login prompt on tty5 (Ctrl+Alt+F5). After you log in and out of tty5, the command should return to the command prompt.
    • Note that the preceding command runs tty5 only once, meaning that if you log in and log out, tty5 does not reprint a login prompt nor accept user input.
    • Note that the preceding command might act funny if runit is already supervising tty5, and may not accept a password. In that case, just keep trying till the screen erases and you have just one login prompt, at which time the preceding command terminates and returns to the command prompt.
  5. Run the run service's run script (/etc/sv/getty-5/run, for instance), as root, at the command prompt. If it doesn't produce a login prompt when you press Ctrl+Alt+F5, troubleshoot until it does. 
    • [root@openrc ~]# /etc/sv/getty-5/run
    • The preceding command should run without returning, and should throw up a login prompt on tty5 (Ctrl+Alt+F5). After you log in and out of tty5, the command should return to the command prompt.
    • Note that the preceding command runs tty5 only once, meaning that if you log in and log out, tty5 does not reprint a login prompt nor accept user input.
    • Note that the preceding command might act funny if runit is already supervising tty5, and may not accept a password. In that case, just keep trying till the screen erases and you have just one login prompt, at which time the preceding command terminates and returns to the command prompt.
  6. runsv /etc/sv/getty-5
    • Note that the preceding command runs tty5 only once, meaning that if you log in and log out, tty5 does not reprint a login prompt nor accept user input.
    • Before continuing this troubleshooting ladder, make sure to terminate tty5 either by pressing Ctrl+C on the runsv command, or logging into and out of tty5.
    • Note that the preceding command might act funny if runit is already supervising tty5, and may not accept a password. In that case, just keep trying till the screen erases and you have just one login prompt, at which time the preceding command terminates and returns to the command prompt.
    • This was the final step on the troubleshooting ladder that should run tty5 only once. All further steps, if everything is functioning correctly, rerun tty5 over and over again, which means you can log in and log out of tty5 as often as you want.
  7. ln -s /etc/sv/getty-5 /service/getty-5
    • The preceding command should run the service within five seconds, and should rerun it every time it ends. That rerunning means that you can log into tty5 and log out, and you'll get another login prompt. If you don't get another login prompt, it's not rerunning the service.
  8. svstat /service/getty-5
    • Rerun the preceding command several times. Output should look like:
      • /service/getty-5: up (pid 1051) 12386 seconds, stopped
      Pay no attention to the "stopped" at the end. As you repeatedly run it, it should be "up" instead of "down" every time, and unless you logged into and out of tty5, it should display the same PID and an ever increasing time.
    • If, every time you run the command without logging into and out of tty5, it keeps having an uptime less than 5 seconds and a different PID, this indicates that this service's run script is failing. Use the sv down /service/getty-5 command to disable the service, then go back to running that run script on the command prompt in order to figure out why it runs at the command prompt but not from within runit.
    • Every time you log out and back in, what runit should do is rerun the service at a different PID, and start from up-time 0 again.
  9. Use the sv up /service/getty-5 and sv down /service/getty-5 commands to enable and disable the tty5 service. View the results with the ps command and the svstat /service/getty-5 command.
  10. Make sure that /etc/runit/1 has the proper commands in it. It should look something like the following:
      #!/bin/sh
PATH=/command:/sbin:/bin:/usr/sbin:/usr/bin

/etc/init.d/rcS
/etc/init.d/rmnologin

touch /etc/runit/stopit
chmod 0 /etc/runit/stopit
    • The rmnologin is not a part of and is not supplied by runit. Just make your own. This command deletes the /etc/nologin file. On my system, that's all it does. If your system doesn't have the rmnologin shellscript, just make a shellscript that deletes /etc/nologin.
    • The rcS command is not a part of and is not supplied by runit. I simply made my own do-nothing shellscript to eliminate the error message.
  11. Make sure that /etc/runit/2 is as supplied by runit. It should look something like this: 
      #!/bin/sh

PATH=/command:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin

exec env - PATH=$PATH \
runsvdir -P /service 'log: ..............................................................
.........................................................................................
.........................................................................................
.........................................................................................
..................................................................'
    • WARNING: Do not copy and paste the preceding. I put linefeeds in the series of dots to accumulate a browser. In the actual /etc/runit/2 shellscript, there should be a series of 400 dots without any linefeeds.
    • If you have an unusual path on your computer, you can accommodate it by changing the PATH= command in this shellscript.
    • The runsvdir command polls every subdirectory in the directory named by its argument (/service in this example), and runs each. It keeps polling every directory, and reruns any that crash.
  12. When you've gotten all the preceding to work, it's time to test it by rebooting. At the grub prompt's kernel line, to the end append a space and then /sbin/runit-init so that this boot boots runit. If the boot goes badly, you can always go back to the system's current default boot.

Using the preceding troubleshooting ladder enables the majority of your diagnostic tests to take very little time. Although I normally recommend Divide and Conquer (half splitting) as the most effective troubleshooting strategy, in this case the ladder takes maybe ten extra minutes if everything was already set to work, but can save hours if there was a problem.

Packaging System Limitations

Hey, I like packaging systems. Without them, I'd have never installed Linux. I typically use over a hundred user applications, and I wouldn't want to manually compile each one. If package managers weren't huge timesavers, I'd have been using FreeBSD years ago. But the last time I installed FreeBSD, it had several conflicting, surprising and error prone package managers. In fact, surprising or ineffective packaging systems are the #1 reason I reject distros and BSD versions. The reason I've stuck with Ubuntu and Debian for seven years was because their apt-get based package management just worked 95% of the time. In other words, a good package manager is worth its weight in gold.

But package managers aren't optimal for every situation.

The most obvious situation I ran across was when I was a developer for the VimOutliner project. We had great install instructions, and for those not wanting to read and follow instructions, we had a trivial install shellscript. It just worked. But various distros packaged VimOutliner, and some of those packages took liberties with the software, including changing some timesaving keystrokes to time-consuming keystrokes. And besides, the packages often didn't work, and when confused users emailed us for help, we couldn't help, because we didn't know what the distro was doing with our software.

Gnome needs a package. Who in their right mind will keep on ./configure;make;make install, time after time, while installing its numerous dependencies. It's nice to have a package for Vim: I'd rather install Vim in five minutes than spend hours tracking down all the dependencies and figuring the root cause of C language errors.

But there are occasional pieces of software so important that you'll hand compile to get the software's full feature set, get the latest version, or just get it to work at all. I don't think I can count on my fingers the number of times I've hand compiled LyX myself: LyX was mission-critical for my business, and when I needed a version with a needed feature, or a version not having the bug that the package-bestowed LyX had, I hand compiled. No biggy: It was important.

Another piece of software I always hand-compile is daemontools. Ubuntu and Debian just couldn't document its installation and use like the program's author, djb, did. So on Ubuntu and Debian, it just took too much trial and error and experimentation to do it right. By doing it djb's way, I could install it on Ubuntu, Debian, OpenSuSE, OpenBSD, or anything else.

I think recent events in the world of Linux have elevated init systems to "compile it yourself" importance. If you want a specific init system (or don't want one specific init system), your choice of init systems mustn't be limited by the packages offered by the distro. Packages get broken, packages get dropped. Runit is particularly suited to hand compilation and installation, because runit is so simple, and has so few compile dependencies, that it's probably easier to compile and install it via Gerrit Pape's instructions.

One place I always like to do an end-run around the packaging system is when I disable or replace processes. For instance, I'm more of a startx kinda guy than one who boots to GUI and types in his password. A few years ago I was a Ubuntu user, and if you know Ubuntu, you know that startx is contrary to their whole philosophy, and it's very difficult to eliminate the display manager via the package manager, in order to get the chance to run startx. No problem, I rigged a lightdm config file to exit immediately, so my computer booted to a command prompt. Another example: On Manjaro/OpenRC, when I wanted to boot to CLI, I didn't use the package manager to remove xdm, I just removed its symlink out of /etc/runlevels/default. I can't count the number of times I've renamed something to keep it from running, or swapped in a symlink to a different executable to run something else. Sure, people criticize me for this, but it gives me lots more time to do real work.

Bottom line: Sometimes it's better to go around the packaging system than to jump through all its hoops, and hope that everything won't change with the next release. Which brings up one last topic...

If your experience is anything like mine, the distro mailing list throws rotten tomatoes at you when they find out you've been doing end runs around the packaging system. When you ./configure;make;make install they become indignant, sometimes demanding you make a package out of your hand compile before installing it. Oh, and if they find out you renamed something to disable or replace it, they go on the warpath, telling you just how completely and utterly wrong you are.

Ignore them.

Here are the facts: It takes 5 seconds to rename something. It takes a couple minutes to do it with the package manager. But the operant fact is this: It takes hours to research exactly which packages you need to install or uninstall. They now have packages to uninstall other packages, for gosh sake.

So, if you install runit and don't want to mess with packaging or mess with the dangerous and confusing grub2, you can always do this:

[slitt@mydesk ~]$ cd /path/to/systemd
[slitt@mydesk ~]$ mv systemd systemd.org
[slitt@mydesk ~]$ cp -p /sbin/runit-init systemd

Is the preceding kludge a stain on my technical ability, or a stain on packaging systems and grub2? I'll leave that question as a challenge for the reader, and I'll simply add that the kludge takes less than a minute.

Another audience who throws rotten tomatoes are the anti-systemd purists. They'll tell you the only true solution is to exile all systemd-related software from your machine, and that requires package management. I completely understand their viewpoint: I've exiled every KDE library from my computers. But KDE doesn't have its tentacles in everything from window manager geegaws to PID1. What I would tell the anti-systemd purists is this: The perfect is the enemy of the good. If you start by package-manager removing everything systemd, as a practical matter you'll be patching things up for the next several days or weeks. If you simply leave the systemd stuff where it lays, but replace its PID1 duties with another init system, then later, one by one, you can replace other systemd components. Either way, it takes a long time to have a systemd free computer. But if you do it my way, you have an operational computer the whole time, plus you really know the internals of your computer, so the package management crowd can't keep making your life difficult with an ever-lengthening parade of init system "improvements".

Manjaro's Pretty Cool

Manjaro's my kind of desktop Linux. It installs easily, like Debian. It has a good package manager, called pacman, that keeps the OS and all but the most customized or rare applications running well. It has respect for those who want systemd, and those who want OpenRC, and makes either pretty easy to use. Because Manjaro is a derivative of Arch, most of the Arch documentation applies, and the Arch documentation is known far and wide to be outstanding.

Speaking of Arch, Manjaro's a descendent of Arch, with these differences:

Out of the box, I found Manjaro 8.11 OpenRC edition to feel pretty much like Debian Wheezy. Easy install, "Just Works", great package manager. The one difference is that Manjaro is a rolling release, with the advantage that you never need to version-upgrade, and the disadvantage that a bad update can leave your computer a brick. When using the pacman packager, get very acquainted with the --ignore option. pacman refuses to upgrade anything if even one package is defective (for instance, bad signature), so when such things go wrong, use --ignore to get the majority upgraded, and then in your leisure address the bad package.

WARNING:

When you use rolling releases, you need to back up all the way to the bare metal, because OS re-installation will be a nightmare.

I haven't seen too much of the overall Manjaro community, but I'm quite pleased with the OpenRC section of the Manjaro community.

A Few Manjaro Tips

Manjaro's a very special distro, because although it comes with systemd, it's relatively easy to replace the systemd with a sysvinit->OpenRC combination. And unlike many distros, its developers aren't hostile to any particular init system. In fact, some Manjaro people have put together a Manjaro/OpenRC DVD image, making it trivial to install an OpenRC equipped Manjaro. It's available here:

https://forum.manjaro.org/index.php?topic=19215.0

I suggest you pay particular attention to the following three things listed as missing on the ISO's installer:

  1. Hostname set via installer probably wont work, can be set in /etc/conf.d/hostname
  2. Console keymap can be set in /etc/conf.d/keymaps
  3. /tmp can be added as tmpfs in /etc/fstab

As examples, here are the examples from my computer:

  1. hostname="littmanj"
  2. keymap="us"
    • Note: keymaps file contains some other fairly obvious config options
  3. tmpfs /tmp tmpfs nodev,nosuid 0 0

Download and burn the ISO and install the DVD, using "Use whole disk" so you have only one partition, fix up the three items listed above, and if your experience is anything like mine, you'll have a very nice working distro in which the kernel passes control to a sysvinit PID1, which promptly yields control to OpenRC.

Now, with a working computer, you can quickly experiment.

Debian's Seen Better Days

These are tough times for Debian. Just now, after all these months, the systemd wars on their user mailing list are winding down. I think the major reason for the relative calm is that most of us who objected to systemd have gone to other distros, or rolled our own. The systemd patriots on the Debian lists probably consider this a good thing, and for those who departed, it is. For Debian, I have a feeling that long term, this exodus of people with the desire and tech chops to control their own computers will dramatically reduce both Debian's mindshare and credibility.

The project really shouldn't have hidden behind their constitution, their CTTE, GR, mailing list rules, "upstreams", arcane package manager commands, and all that stuff. They shouldn't have made these things the user's problem. Because, in the end, a lot of us wanted a working system with interchangeable parts, not project baggage.

The systemd patriots on the list were forever characterizing those of us wanting choice (and obviously systemd is built from the ground up to limit choice) as "haters", "offtopic", "spammers" and worse. They told us that if we didn't like it, we were free to make our own.

So we did.

Some of us are de-systemding Debian, some of us have moved to other distros, and some of us are making software rendering systemd moot on any distro. All these alternatives are excellent, and I want to make it clear that any ill feeling I might (or might not) have toward Debian do not extend to the Devuan project (formerly called the "Debian fork") or the things being discussed on the modular-debian mailing list. To the extent possible, I'll work to help any and all de-systemd'd Debian forks or descendants. We're all in the same boat.

The future is a crystal ball. Time will tell. My best guess is that Debian's best days are behind it.

Wrapup, 12/17-12/19/2014

I installed runit on a native sysvinit->OpenRC Manjaro system (native because somebody made an OpenRC edition, normal Manjaro is systemd). At this point, my OpenRC setup runs only devfs, sysfs, and udev, with runit handling the remainder. I'm pretty sure that the only need for devfs and sysfs is because they're required by udev. Once I find runit init scripts for udev, and possibly the other two, I can pretty much have my way with the initialization of any distro. Perhaps it would be fun to initialize Fedora with runit: how's that for irony?

The system I described from 12/17 to 12/19 is nowhere near robust enough for daily work. There are no provisions for edge cases, or even common cases like multiple partitions. It doesn't prophylactically run fsck after X number of boots, and not doing those fscks is dangerous in production.

But that's not the point. The point is that I, a Troubleshooting Trainer with no admin experience, managed to get runit initting a system. Replacing an init system, even without help from the package manager, is not as prodigious a task as many people think, and perhaps not as prodigious a task as other people want you to think. It's going to be documented, by me or by others, very shortly. And then, those of us who don't like systemd's architecture can begin to replace the systemd weld-ons, like udev, with non-systemd replacements.

12/21/2014: RichFelker->OpenRC

A few days ago, Rich Felker licensed his 16 line init program with a free software license (see bottom of page for both license and code). That day I fulfilled a long ambition to boot off it, and it was cool.

First, to get gcc -Wall to work right, I needed to add #include<sys/wait.h> to the list of includes. Then it compiled no errors, no warnings:

[root@openrc ~]# gcc -Wall richfelker.c [root@openrc ~]#

Next, I copied a.out to /usr/bin/init.richfelker. I didn't have an /etc/rc file, so I made one that looks like this:

#!/bin/sh /usr/bin/openrc sysinit /usr/bin/openrc boot /usr/bin/openrc default

Then I booted to it, adding init=/usr/bin/init.richfelker to the end of the Grub2 kernel line.

It booted, with three caveats:

  1. There were no virtual terminals
  2. It shut down badly, requiring a journal restore on every reboot.
  3. It starts without network facilities, so you need to run a shellscript to start the network.

Other than the preceding, it booted perfectly to the display manager xdm. The referenced network-enabling shellscript is the same one I used in the December 17, 2014 section.

Respawn and Virtual Terminals

It turns out that OpenRC has no facility for respawning processes it manages, so OpenRC itself cannot manage virtual terminals. I tried the following OpenRC init script:

#!/usr/bin/openrc-run
# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

depend()
{
        need localmount
        after bootmisc
}

start() {
        ebegin "Starting getty2"
        /usr/bin/start-stop-daemon --start --background \
        --exec /usr/bin/agetty   --make-pidfile \
        --pidfile /run/getty2.pid  -- 38400 tty2 linux
        eend $?
}

reload() {
        ebegin "ReStarting getty2"
        /usr/bin/start-stop-daemon  \
        --exec /usr/bin/agetty   \
        --pidfile /run/getty2.pid -s1
        eend $?
}

stop() {
        ebegin "Stopping getty2"
        start-stop-daemon --stop --exec /usr/bin/agetty --pidfile /run/getty2.pid
        eend $?
}

Close, but no cigar. The preceding runs agetty once, because OpenRC has no native, idiomatic way of respawning a service the way sysvinit and runit and even daemontools can do. I tried putting the getty call in a looping shellscript, but no joy. Some bizarre property of the agetty command reaches back and kills the thing that started it. So if you want virtual terminals spawned in a RichFelker->OpenRC setup, you need to have OpenRC run daemontools, and let daemontools manage the virtual terminals.

Rich Felker's 16 line init program is potentially a great way to start OpenRC, or maybe even a tweaked daemontools or daemontools-encore, but more work needs to be done to make sure such a setup can respawn services to run virtual terminals, shut down cleanly, and automatically run networking. This could be an excellent alternative, because in 2014, I think many of us would prefer to keep sysvinit (and Upstart and systemd) off our computers. If you find new and better ways to use Rich Felker's 16 line init, either by managing what descends from /etc/rc, or by modifying Rich's code, please email me.

12/23/2014: ip Only Network Start Script

A few days ago I made a Network Start Script to start the network in a runit->OpenRC init with minimal use of OpenRC. I was quickly informed that ifconfig is deprecated, and everything should be done with the ip command. So I changed the script, replacing all ifconfig and route commands with their corresponding ip commands. The following is the new, ip only Network Start Script:

#!/bin/sh
hostname -F /etc/hostname
ip link set dev lo up
ip link set dev eth1 down
ip addr add 192.168.100.88/24 dev eth1
ip link set dev eth1 up
ip route add default via 192.168.100.96

[ Training | Troubleshooters.Com | Email Steve Litt ]

12/23/2014: Pure Runit Init Proof of Concept

No systemd. No sysvinit. No OpenRC. No dbus. No package manager. Just pure runit, aided by a shellscript to enable devfs and a shellscript to enable udev. Boots Grub menu to command prompt in 8 seconds, shuts down from LXDE to poweroff in 5. Plays Youtube videos, including sound. Due to lack of dbus, it can't run a display manager (xdm, lxdm, gdm etc) but startx starts LXDE just fine. The /etc/runit/1 script follows:

#!/bin/sh
# system one time tasks

PATH=/command:/sbin:/bin:/usr/sbin:/usr/bin

### START devfs
/root/run_devfs.sh

### START sysfs
#  TODO, but seems not to matter

### START udev 
/root/udev_start.sh

### ENABLE SOUND MIXING WITHOUT ALSA, PULSEAUDIO (USE rexima or aumix)
modprobe snd_mixer_oss

### SET UP NETWORK WITHOUT systemd, NetworkManager, dbus or wicd ###
### SORRY LENNART ###
/root/upnet.sh

/etc/init.d/rcS
/etc/init.d/rmnologin

touch /etc/runit/stopit
chmod 0 /etc/runit/stopit

In the preceding, the following shellscripts take the place of formerly OpenRC-started functionalities:

The remainder of this section discusses each of the shellscripts, and then discusses the true meaning of a pure runit init.

/root/run_devfs.sh

This is one of the ugliest shellscripts I've ever made. Actually, I didn't really make it: I re-molded Manjaro's devfs OpenRC init script into a shellscript. It's ugly, I have no doubt it's fragile, I'd be shocked if it were portable, but it works, and that's a good sign for a proof of concept. The shellscript follows:

#!/bin/sh
# Copyright (c) 2007-2008 Roy Marples <roy@marples.name>
# Released under the 2-clause BSD license.
# Heavily kludged by Steve Litt in 2014

description="Set up the /dev directory"

mount_dev()
{
        local mountopts="exec,nosuid,mode=0755"
        if grep -q devtmpfs /proc/filesystems; then
                devfstype=devtmpfs
                mountopts="$mountopts,size=10M"
        elif grep -q tmpfs /proc/filesystems; then
                devfstype=tmpfs
                mountopts="$mountopts,size=10M"
        fi
        if [ -n "$devfstype" ]; then
                mount -n -t $devfstype -o $mountopts dev /dev
        else
                echo "WARNING  This kernel does not have devtmpfs or tmpfs support, and there"
                echo "WARNING  is no entry for /dev in fstab."
                echo "WARNING  This means /dev will not be mounted."
                echo "WARNING  To avoid this message, set CONFIG_DEVTMPFS or CONFIG_TMPFS to y"
                echo "WARNING  in your kernel configuration or see /etc/conf.d/devfs"
        fi
}

seed_dev()
{
        # Seed /dev with some things that we know we need

        # creating /dev/console, /dev/tty and /dev/tty1 to be able to write
        # to $CONSOLE with/without bootsplash before udevd creates it
        [ -c /dev/console ] || mknod -m 600 /dev/console c 5 1
        [ -c /dev/tty1 ] || mknod -m 620 /dev/tty1 c 4 1
        [ -c /dev/tty ] || mknod -m 666 /dev/tty c 5 0

        # udevd will dup its stdin/stdout/stderr to /dev/null
        # and we do not want a file which gets buffered in ram
        [ -c /dev/null ] || mknod -m 666 /dev/null c 1 3

        # so udev can add its start-message to dmesg
        [ -c /dev/kmsg ] || mknod -m 660 /dev/kmsg c 1 11

        # extra symbolic links not provided by default
        [ -e /dev/fd ] || ln -snf /proc/self/fd /dev/fd
        [ -e /dev/stdin ] || ln -snf /proc/self/fd/0 /dev/stdin
        [ -e /dev/stdout ] || ln -snf /proc/self/fd/1 /dev/stdout
        [ -e /dev/stderr ] || ln -snf /proc/self/fd/2 /dev/stderr
        [ -e /proc/kcore ] && ln -snf /proc/kcore /dev/core

        # Mount required directories as user may not have them in /etc/fstab
        for x in \
                "mqueue /dev/mqueue 1777 ,nodev mqueue" \
                "devpts /dev/pts 0755 ,gid=5,mode=0620 devpts" \
                "tmpfs /dev/shm 1777 ,nodev,mode=1777 shm" \
        ; do
                set -- $x
                grep -Eq "[[:space:]]+$1$" /proc/filesystems || continue
                mount | grep $2 && continue

                if [ ! -d $2 ]; then
                        mkdir -m $3 -p $2 >/dev/null >&1 || \
                                echo "WARNING  Could not create $2!"
                fi

                if [ -d $2 ]; then
                        echo "Mounting $2"
                        if ! mount | grep $2; then
                                mount -n -t $1 -o noexec,nosuid$4 $5 $2
                        fi
                fi
        done
}

startt()
{
        mount_dev
        seed_dev
}

startt

The preceding is as ugly as it gets, and I bet over half of it is fluff for edge cases. I doubt it would work other places, but it works on my pure runit Manjaro machine. If my pure runit setup has an Achilles Heel, this is it.

My finding was what I need really needed from this script was /dev/pts so that I could ssh into the computer, and so LXDE could run terminals with bash. I left the rest of it in because I didn't know which things were a prerequisite of /dev/pts.

If you try this script and find that it doesn't work for you, or if you need to modify it, please email me so I can document more knowledge on devfs.

/root/udev_start.sh

Five lines and you're done:

#!/bin/sh
/usr/bin/udevd --daemon
/usr/bin/udevadm trigger --action=add --type=subsystems
/usr/bin/udevadm trigger --action=add --type=devices
/usr/bin/udevadm settle

I have absolutely no idea what any of the preceding means: It was given to me by user Subsentient on #epoch on Freenode. Subsentient is the main developer for the Epoch Init System, another one I want to try very soon.

What I know is that the preceding script works on Subsentient's machine, it works on my machine, so it's probably somewhat portable, and if not, it's easy for anyone familiar with the ins and outs of udev to tweak into working.

Thank you, Subsentient!

/root/upnet.sh

I've talked about this shellscript before. I made it, and as far as I know, it's constructed well and will work anywhere the ip command is available, which, as far as I know, is any Linux distro. The shellscript follows:

#!/bin/sh
hostname -F /etc/hostname
ip link set dev lo up
ip link set dev eth1 down
ip addr add 192.168.100.88/24 dev eth1
ip link set dev eth1 up
ip route add default via 192.168.100.96

The preceding shellscript completely removes the need for playpen apps like NetworkManager and wicd, at least if you're dealing with wired Ethernet. As far as Wifi, some time in the not too distant future I'm going to make a NetworkManager workalike that relies on wpa_supplicant, iwlist, ip, and a little sudo fu. No dbus necessary.

Why This Is Important

The ability to lay down a hand compiled and configured runit right alongside a distro's normal init, and simply run runit instead of what the distro had in mind, is huge. It takes the distro's packagers right out of the equation. If your distro decides to reach in and replace your init system, you can simply lay down a pure runit init, back up their /usr/bin/init, and replace it with your own. Or better yet, make a tiny shellscript, so that every time an upgrade replaces /usr/bin/init, you run the shellscript to get runit back. The shellscript would look something like this:

#!/bin/sh
datestamp=`date +"%Y%m%d_%H%M%S"`
cd /usr/bin
cp -p init init_thoseguys$datestamp
cp -p runit-init init

NOTE:

The reason I keep saying /usr/bin/init instead of /sbin/init is because Manjaro is one of those "forward thinking" distros whose /sbin is just a symlink to /usr/bin. Guess whose idea that was? Anyway, runit aborts to a busybox prompt if you init to it in a symlinked directory. So if you have one of those Freedesktop.org inspired distros, just init to /usr/bin/init and be happy.

By the way, this is about the only downside I've found to Manjaro in my three weeks of testing it out.

Once the documentation for hand compiled, hand configured runit becomes accurate, complete with better shellscripts to start devfs, sysfs, udev, and even dbus, as well as /etc/runit/1, it becomes almost trivial to convert almost any distro to runit, always assuming you're not married to Gnome or Unity or KDE or some other "Desktop Environment" that does it all for you. And if you like such a desktop environment, you probably wouldn't mind systemd, so the point is probably moot.

NOTE:

If you like a heavy, integrated desktop environment but don't want systemd, check out the Devuan project. They're not shipping yet, but their distro will have all of Debian's capabilities, without the systemd baggage.

Anyway, with the ability to lay down a clean and simple runit system, you control your computer with an iron hand.

But is this really ready for prime time?

Of course not. Like I said, I have serious doubts about the portability of my /root/run_devfs.sh script. And I'm sure many other problems will pop up moving this procedure from distro to distro. None of which matters.

The point is, sooner or later a critical mass of knowledge will develop about standalone runit, and accurate, easy documentation will follow. What you're reading now is a proof of concept, but soon you'll be able to do this on any distro you want.

12/24/2014: Pure Epoch Init System

The afternoon of 12/24 I finished my research on the Epoch Init System, and ran a "hello world" to boot with an Epoch init and make available a virtual terminal. By that night, I had a full blown, Epoch-initted, Linux desktop machine running on my Manjaro Experimental Computer. It worked, almost out of the box. One of the reasons I got it running so fast is that, of all the init systems, Epoch has some of the best documentation:

Main Page http://universe2.us/epoch.html
Site map Bottom of http://universe2.us/epoch.html
Config file howto http://universe2.us/epochconfig.html
Epoch commands http://universe2.us/epochmanagement.html
FAQ http://universe2.us/epochfaq.html
Downloads Bottom of http://universe2.us/epoch.html
Some misc resources http://universe2.us/collector/
Sample epoch.conf http://universe2.us/collector/epoch.conf

On most inits, the hardest thing to do is formulate the init scripts or the, for want of a better word, "unit files". I guess you could say that Epoch's config more resembles "unit files", but they're all in one file, /etc/epoch/epoch.conf. One huge benefit of Epoch is its author, a guy named Subsentient, has provided a sample epoch.conf. In fact, it's one he uses to init Fedora. Think about that for a second. Anyway, the sample epoch.conf is located at http://universe2.us/collector/epoch.conf. That sample file clears up a lot of questions you'll have at first, and contributes to quick init replacement.

Among all the init systems I've found so far, Epoch has the best documentation. It's also extremely simple, and it's a very easy replacement for your current init.

Perhaps I should define "replacement". Some folks, let's call them the Purists, define it as removing every vestige of the old init, and replacing it with the new init. It's like ripping up the old highway, and paving a new, better highway right over the site of the old one. That's not how I use Epoch (or runit).

The other definition of "replacement" is building a better highway in parallel with the first, perhaps even using portions of the former highway in the route, and just changing the signs so that people follow the new route. This is the sense in which I replaced my current init with Epoch (same's true for runit). My way's quicker and easier for a bunch of different reasons:

Bottom line, I'm a huge fan of laying down a parallel init system and using that. Much easier, much faster, no package manager interference.

Another advantage of Epoch is its reasonably simple service configurations. Epoch's config is declarative like systemd. Once you learn the ins and outs from the docs and sample epoch.conf, it's fairly easy. I'm not saying scripts are guaranteed to be difficult: runit scripts are simple. But one look at sysvinit scripts or OpenRC scripts is enough to make you run screaming, and I think the #1 reason for resistance to various init systems is the difficulty of configuring individual services. Bottom line, unlike sysvinit or OpenRC, you don't need to be a professional Admin to perform Epoch service config: A mere civilian can handle the job quite well.

So which is better, Epoch or runit? I don't know, which would you prefer, having your robot win the 220 pound combat title at Robogames, or driving a Tesla controlled by your hand programmed Raspberry Pi? Some choices are just too good to be contemplated. In making such a decision, I'd look at a few guidelines:

I'm stoked about the Epoch Init System because it's a replacement init that a fairly smart computer user can use to replace his current init in less than a day. Let others argue the pros and cos of systemd: Systemd won't darken your door again after that day of setting up Epoch.

All init systems are somewhat complex, some more than others. To maximize your likelihood of a straight forward route to an Epoch initted computer, I've written the next section, called Getting Epoch Running...

Getting Epoch Running

This section is long for one and only one reason: I've written it explicitly that someone who has never before installed and configured an init system can do so, with a minimum of ambiguity, confusion or surprise. Its length has nothing to do with it being complicated: In fact, it's the simplest and fastest to install and configure init system I've to date, and also the best documented. So let's get started...

First, let me repeat the official Epoch documentation pages.

Main Page http://universe2.us/epoch.html
Site map Bottom of http://universe2.us/epoch.html
Config file howto http://universe2.us/epochconfig.html
Epoch commands http://universe2.us/epochmanagement.html
FAQ http://universe2.us/epochfaq.html
Downloads Bottom of http://universe2.us/epoch.html
Some misc resources http://universe2.us/collector/
Sample epoch.conf http://universe2.us/collector/epoch.conf

The preceding official documentation pages are spectacularly helpful. Use them!

Starting on 12/24/2014, and continuing until there's a newer stable version, the version you want to download is 1.2.0, code name "Peroxide". Make yourself a blank directory (I used /root/build), and within that directory untar the tarball. The following is what you get, except that before you compile, the built and objects directories are empty:

build
`-- epoch-1.2.0
    |-- buildepoch.sh
    |-- built
    |   |-- bin
    |   |   `-- wall -> ../sbin/epoch
    |   `-- sbin
    |       |-- epoch
    |       |-- halt -> ./epoch
    |       |-- init -> ./epoch
    |       |-- killall5 -> ./epoch
    |       |-- poweroff -> ./epoch
    |       |-- reboot -> ./epoch
    |       `-- shutdown -> ./epoch
    |-- CHANGELOG
    |-- objects
    |   |-- actions.o
    |   |-- config.o
    |   |-- console.o
    |   |-- main.o
    |   |-- membus.o
    |   |-- modes.o
    |   |-- parse.o
    |   `-- utilfuncs.o
    |-- README
    |-- src
    |   |-- actions.c
    |   |-- config.c
    |   |-- console.c
    |   |-- epoch.h
    |   |-- main.c
    |   |-- membus.c
    |   |-- modes.c
    |   |-- parse.c
    |   `-- utilfuncs.c
    `-- UNLICENSE.TXT

Observe the following command:

buildepoch.sh

The preceding command fills in the built and objects trees. If you want to use special arguments to buildepoch.sh, that's doable, but that's getting beyond the scope of this section. Anyway, after running buildepoch.sh, built/sbin/epoch is the Epoch init executable. Copy it wherever you want, but be sure to back up the old version of anything you copy it over. I copied the file temporarily to /e, because I wanted an easily typeable name to put in Grub2's kernel line's init=/e variable.

But don't boot yet. To tell the Epoch init system what to do, you need to configure /etc/epoch/epoch.conf. I started with the Sample epoch.conf file, commented out most of it, uncommented only stuff I absolutely needed, one at a time. Here's the /etc/epoch/epoch.conf I now use to boot up my Manjaro computer to a reasonable desktop computer, with an LXDE window manager and ability to play Youtube videos:

BootBannerText=Sorry, Lennart!
BootBannerColor=CYAN

Hostname=FILE /etc/hostname
DefaultRunlevel=boot
EnableLogging=true
DisableCAD=true
BlankLogOnBoot=true
MountVirtual=procfs sysfs devpts+ devshm+

# DEFINE PRIORITY CONSTANTS
DefinePriority=Rwfs_Start 30
DefinePriority=Early_Getty_Start 40
DefinePriority=Sysclock_Start 35
DefinePriority=Udev_Start 20
DefinePriority=Network_Start 50
DefinePriority=Dev_Mixer_Start 60
DefinePriority=Typical_Daemon_Start 80
DefinePriority=Typical_Getty_Start 70
DefinePriority=Display_Manager_Start 120


DefinePriority=Display_Manager_Stop 40
DefinePriority=Typical_Daemon_Stop 10
DefinePriority=Sysclock_Stop 80
DefinePriority=Killall5_Soft_Stop 90
DefinePriority=Killall5_Stop 91
DefinePriority=Rwfs_Stop 100 



ObjectID=getty2
        ObjectDescription=Early getty on /dev/tty2
        ObjectStartCommand=agetty tty2 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Early_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

# MAKE readwrite just after udev instantiates,
# MAKE readonly again after all killalls
ObjectID=rwfs
        ObjectDescription=root filesystem read-write support
        ObjectStartCommand=/usr/bin/mount -o remount,rw /
        ObjectStopCommand=/usr/bin/mount -o remount,ro /
        ObjectStartPriority=Rwfs_Start
        ObjectStopPriority=Rwfs_Stop
        ObjectEnabled=true
        ObjectRunlevels=boot 

# KINDLY request all processes to end themselves
# Give them 2 seconds to do so.
ObjectID=killall5_soft
        ObjectDescription=Terminating all processes
        ObjectStopCommand=killall5 -15 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=Killall5_Soft_Stop
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

# MURDER all remaining processes
# Give them 2 seconds to do so.  
# Then hand control to service rwfs to 
# remount / read-only to prevent filesystem corruption     
ObjectID=killall5
        ObjectDescription=Killing all processes
        ObjectStopCommand=killall5 -9 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=Killall5_Stop
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

ObjectID=sysclock
        ObjectDescription=Configuring system clock
        ObjectStartCommand=hwclock -s
        ObjectStopCommand=hwclock -w
        ObjectStartPriority=Sysclock_Start
        ObjectStopPriority=Sysclock_Stop
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN simple shellscript to get udev running
# Udev is necessary to start network
# Run it before remounting / read-write.
# Don't bother to stop it at all
ObjectID=udevd
        ObjectDescription=Starting udev
        ObjectStartCommand=/root/udev_start.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=Udev_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN shellscript to set up network. Works on any distro having
#  the ip command. No config files necessary.
#  start it after Udev is up and / is read-write 
#  and the early Getty is running
ObjectID=network
        ObjectDescription=Setting up net devices and the network
        ObjectStartCommand=/root/upnet.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=Network_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# LOAD kernel module for /dev/mixer, without ALSA or PULSE
# Run after early Getty and network, but before daemons
# No need to stop it on shutdown.
ObjectID=devmixer
        ObjectDescription=Setting up /dev/mixer via oss
        ObjectStartCommand=/usr/bin/modprobe snd_mixer_oss
        ObjectStopCommand=NONE
        ObjectStartPriority=Dev_Mixer_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

ObjectID=getty3
        ObjectDescription=Getty on /dev/tty3
        ObjectStartCommand=agetty tty3 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty4
        ObjectDescription=Getty on /dev/tty4
        ObjectStartCommand=agetty tty4 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty5
        ObjectDescription=Getty on /dev/tty5
        ObjectStartCommand=agetty tty5 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty6
        ObjectDescription=Getty on /dev/tty6
        ObjectStartCommand=agetty tty6 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

# MANAGE sshd daemon
ObjectID=sshd
        ObjectDescription=Manage sshd daemon
        ObjectStartCommand=/usr/sbin/sshd
        ObjectStopCommand=PIDFILE /run/sshd.pid
        ObjectStartPriority=Typical_Daemon_Start
        ObjectStopPriority=Typical_Daemon_Stop
        ObjectEnabled=true
        ObjectRunlevels=boot
        ObjectOptions=SERVICE AUTORESTART

# MANAGE crond daemon
ObjectID=crond
        ObjectDescription=Manage crond daemon
        ObjectStartCommand=/usr/bin/crond
        ObjectStopCommand=PIDFILE /run/crond.pid
        ObjectStartPriority=Typical_Daemon_Start
        ObjectStopPriority=Typical_Daemon_Stop
        ObjectOptions=SERVICE AUTORESTART
        ObjectEnabled=true
        ObjectRunlevels=boot

# RUN ONCE Display Manager.
# Runlevels=boot to enable, Runlevels=gui to disable to GUI via startx
# NOTE: Display Manager might invoke dbus.
# NOTE: Toggle ObjectEnabled to turn GUI booting on or off.
ObjectID=DisplayManager
        ObjectDescription=DisplayManager
        ObjectPIDFile=/run/DisplayManager.pid
        ObjectStartCommand=/usr/bin/lxdm &
        ObjectStopCommand=PID
        ObjectStartPriority=Display_Manager_Start
        ObjectStopPriority=Display_Manager_Stop
        ObjectEnabled=true
        ObjectRunlevels=boot

A few comments on the preceding /etc/epoch/epoch.conf:

The Process

The preceding subsection mentioned the preference of reaching a working system using a process rather than just copying and pasting from this web page. That's because your setup will almost certainly be different than mine, so my epoch.conf probably will fail on your system. Here is the high level of the process I recommend:

  1. Hello World: Shell-Only Init
  2. Hello World: Getty-Only Init
  3. Add in Killall Processes
  4. Add in Sysclock
  5. Run Network
  6. Make Epoch Your Default Init
  7. Add in /dev/mixer Capability
  8. Add in Gettys
  9. Add in Daemons
  10. Convert priority numbers to constants
  11. Recap Your Results
  12. Optional: Add in Display Manager

Hello World: Shell-Only Init

Any complex learning situation begins with a Proof of Concept, sometimes called a POC or a "Hello World". The purpose of a Hello World is to get the new technology to run in its simplest form, so simple as not to be useful. No error messages, no use case adaptations, nothing but some feedback that yes, it ran. To skip the "Hello World" phase when starting a project with an unknown technology is to trudge the Highway to Heartache.

As your "Hello World", you're going to simply use the Epoch Init System to boot into /sbin/sh or /usr/bin/sh.

NOTE:

Luminaries in the world of Linux have seen fit to stop the decades-old split between programs needing to run before mounts have been completed, and ordinary programs, and thus have made /sbin and /bin symlinks to /usr/bin.

[root@openrc ~]# ls -ldF /sbin
lrwxrwxrwx 1 root root 7 12.07.2014 05:57 /sbin -> usr/bin/
[root@openrc ~]# ls -ldF /bin
lrwxrwxrwx 1 root root 7 12.07.2014 05:57 /bin -> usr/bin/
[root@openrc ~]#

Hey, don't blame me, I'm just reporting the situation.

Anyway, when /bin is a symlink, that symlink might not be established early in the init process, so use the real hard location, /usr/bin/sh. If /bin is a completely different directory, use /bin/sh.

Follow these steps:

  1. Copy your built/sbin/epoch to /e. This makes it much easier, quicker, and more error resistant to, at the Grub menu, edit it to append init=/e. I know that putting files in the root directory is normally a worst practice, but when doing experimentation and troubleshooting, fast cycles and error resistance mean the world. When the system is stable, you can move /e to /usr/bin/init or /sbin/init where it belongs.
  2. Copy your built/sbin/epoch to /usr/bin/epoch. Once you've done that, you can use the epoch reboot and epoch poweroff commands to reboot and power down your computer. Please remember that without this step, none of the epoch commands described later in this document will work.
  3. Create the following /etc/epoch/epoch.conf: 
    BootBannerText=Sorry, Lennart!
BootBannerColor=CYAN

Hostname=FILE /etc/hostname
DefaultRunlevel=boot
EnableLogging=true
DisableCAD=true
BlankLogOnBoot=true
MountVirtual=procfs sysfs devpts+ devshm+

# SHELL sh hello world
ObjectID=sh
        ObjectDescription=Hello world via sh
        ObjectStartCommand=/usr/bin/sh
        ObjectStopCommand=NONE
        ObjectStartPriority=1
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectRunlevels=boot
  4. Observe that the ObjectStartCommand does not end in an ampersand. This command is not meant to run in the background, and won't deliver a command prompt if run in the background.
  5. Observe the line starting with ObjectID. This begins a service, and the comment line before it pertains to that service. The service in this case simply runs the shell, /bin/sh or /bin/sh
  6. Reboot the machine.
  7. At the Grub prompt, press e to edit the boot, and add the following string to the end of the kernel line: 
    init=/e
  8. The machine should quickly bring you to a shell prompt, where you can do commands like ls and mount.

    NOTE:

    This environment is nowhere near a complete one. You'll see warnings complaining about ioctl and warnings. You cannot correctly run the epoch command in this environment, so Ctrl+Alt+Del is the only way you can reboot from this environment.

  9. If the preceding steps did not bring you to a command prompt, troubleshoot.
  10. Understand that you have now proven that the plumbing of the Epoch Init System works on your computer. Your next step is a Hello World that actually creates a Virtual Terminal to log into.

You're done with this experiment. Use Ctrl+Alt+Del to reboot your computer, and let it boot into its normal, functional init.

Hello World: Getty-Only Init

  1. Experiment with your system to determine exactly how to start a virtual terminal. Depending on your distro, it might use getty, agetty, mingetty, getty or something else. Experiment until you have the exact command, with full path, required to throw up a login on Ctrl+Alt+F9. For instance, on my Manjaro system, it's 
    exec /usr/bin/agetty 38400 tty9 linux
    or 
    exec /usr/bin/agetty tty9
    or 
    nohup /usr/bin/agetty tty9 &
    It's vital you complete this step before going on. This might not be the exact command used in Epoch, and agetty is really weird, but without knowing how to make a getty at the command prompt, you're dead meat on the next step.
  2. Make and boot to a getty-only /etc/epoch/epoch.conf. Experiment with it until it works. Here's a hint: The getty line will probably look something like the following: 
    ObjectStartCommand=agetty tty2 &
  3. Within your /etc/epoch/epoch.conf replace entire Shell service with a Getty service, which should look something like the following: 
    ObjectID=getty2
        ObjectDescription=Early getty on /dev/tty2
        ObjectStartCommand=agetty tty2 &
        ObjectStopCommand=NONE
        ObjectStartPriority=40
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot
  4. The ObjectStartPriority is fairly high, because other services, such as Read Write File System and Udev, need to happen before it.
  5. In the preceding step, if agetty isn't available on your computer or it doesn't work on your computer, substitute one of the getty commands that worked during your previous experimentation on the command line.
  6. Reboot, and append init=/e (the location you copied the epoch executable) to the kernel line.
  7. When the boot messages stop coming, press Ctrl+Alt+F2 to access tty2. If you can repeatedly log in as both your username and root, you now have a working proof of concept (Hello World) Epoch Getty implementation. If not, troubleshoot.
  8. When you're able to repeatedly log into Ctrl+Alt+F2 as both your username and root, you've proven the concept of creating a Getty. Your next step will be to add code to run on reboot or shutdown.

Add in Killall Processes

The closer you get to making everything exit gracefully, the less likely file corruption or other data anomalies become. Add the following services to the bottom of your /etc/epoch/epoch.conf:

# MAKE readwrite just after udev instantiates,
# MAKE readonly again after all killalls
ObjectID=rwfs
        ObjectDescription=root filesystem read-write support
        ObjectStartCommand=/usr/bin/mount -o remount,rw /
        ObjectStopCommand=/usr/bin/mount -o remount,ro /
        ObjectStartPriority=30
        ObjectStopPriority=100
        ObjectEnabled=true
        ObjectRunlevels=boot 

# KINDLY request all processes to end themselves
# Give them 2 seconds to do so.
ObjectID=killall5_soft
        ObjectDescription=Terminating all processes
        ObjectStopCommand=killall5 -15 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=90
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

# MURDER all remaining processes
# Give them 2 seconds to do so.  
# Then hand control to service rwfs to 
# remount / read-only to prevent filesystem corruption     
ObjectID=killall5
        ObjectDescription=Killing all processes
        ObjectStopCommand=killall5 -9 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=91
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

Notice that the two Killall services happen on shutdown only. Notice also the order of the three added services. Service rwfs, which remounts read-write on startup and remounts read-only on shutdown, starts before tty2 so that tty2 logging works. However, to prevent damage by rogue device drivers, when Udev is added later on, it will happen before service rwfs, so that if the system is booted read-only, Udev won't be able to write the filesystem.

The shutdown order is especially interesting. With ObjectStopPriority of 100, the rwfs service is intentionally the last to happen. Two steps before that, service killall5_soft executes first, sending every process a SIGTERM and waiting 2 seconds for everything to terminate itself. Then, service killall5 sends every remaining process a SIGKILL (kill -9) to force their immediate termination, and waits 2 seconds for them to die. Finally, service rwfs sets the file system read-only, and Epoch either powers off or reboots, depending on the runlevel passed to the epoch command.

At this point you have a bootable system that's reasonably considerate to your filesystems. No network, no drivers, you can't ssh into it, but it works.

Add in Sysclock

The hardware clock should start up after Udev but before rwfs. So we'll give it a priority of 35. Add the following to the bottom of your existing /etc/epoch/epoch.conf:

ObjectID=sysclock
	ObjectDescription=Configuring system clock
	ObjectStartCommand=hwclock -s
	ObjectStopCommand=hwclock -w
	ObjectStartPriority=35
	ObjectStopPriority=80
	ObjectEnabled=true
	ObjectOptions=RAWDESCRIPTION
	ObjectRunlevels=boot

The preceding turns on the hardware clock. Note that the hardware clock should start just after Udev and just before rwfs. It should stop late, but before Killall5_soft.

Adding the sysclock service probably won't have especially striking effects, but it's part of your timekeeping system. After using tty2 to make sure everything's OK, go on to the next step.

Run Network

Friends whose opinions I trust tell me that ifconfig is deprecated, and to use the ip command for everything. So that's what this section does.

With the system booted as indicated in the Add in Sysclock sub-subsection, perform the following command:

ip addr | cut -b 1-60

It should show lo and any eth devices, but instead it shows just the lo device, because no network device drivers are loaded. Not only that, the lack of output from the following command proves that the lo device isn't even up:

ip link show up

This sub-subsection gets a wired Ethernet network all set up, and does so without respect to the silly config files that every distro puts in a different place with a different name. In order to run an Ethernet card, you need the following two events:

  1. Load the network card's device drivers
  2. Run an ip command based shellscript to set up lo, eth?, and hostname.

The way I did it was to run Udev, which loaded the right drivers. This has the advantage that I needn't know driver names or details. It has the disadvantage that Udev is systemd code. In this case, I chose the easy way. To run Udev, I use a script which is fairly simple but seems fairly universal. The script, called /root/udev_start.sh, follows:

#!/bin/sh
/usr/bin/udevd --daemon
/usr/bin/udevadm trigger --action=add --type=subsystems
/usr/bin/udevadm trigger --action=add --type=devices
/usr/bin/udevadm settle

After running the preceding shellscript, the ip addr command shows the Ethernet card(s) too. Now all that's required is to configure the network. The following is a distro-independent way to configure your ethernet address to 192.168.100.87. Notice that on my Manjaro computer, eth1 is configured to 192.168.100.87, and eth0 has no Ethernet cable plugged into it. The script, called /root/upnet.sh, follows:

hostname -F /etc/hostname
ip link set dev lo up
ip link set dev eth1 down
ip addr add 192.168.100.87/24 dev eth1
ip link set dev eth1 up
ip route add default via 192.168.100.96

After running the preceding script, you now have a fully functional wired Ethernet network at 192.168.100.87 with gateway 192.168.100.96. You will probably need to change IP addresses to fit your network, but that's OK. If networking works, you're done. If not, troubleshoot.

Troubleshooting

Start by seeing how much or how little network access you have. Assuming you're configured for the 192.168.100.0/24 subnet, try pinging a runninc computer on that subnet. If any of the pings succeed, you know at least your Network card works, and is connected to your LAN.

Next, ping known good Internet address 8.8.8.8. If you succeed, you know you're getting out through your default route to the Internet, and any problems are mere DNS. If the ping to 8.8.8.8 fails, make sure your computer's default route is set to the LAN address of your router. If they are the same, take a look at your router/firewall.

If you can ping 8.8.8.8, but can't ping various Internet URLs such as www.troubleshooters.com, you have a probable DNS failure. Look at /etc/resolv.conf. Back it up, and then replace it with the following pointer to Google's public DNS servers:

nameserver 8.8.8.8
nameserver 8.8.4.4

Once shellscripts /root/udev_start.sh and /root/upnet.sh produce a completely working network, the next step is to make sure they run on boot. Do this by making them Epoch services...

Yes, I Know

Yes, I know that this setup guide hard coded the IP address, completely missing both wired DHCP and Roaming Wifi use cases. The former is accomplished with the dhcpd daemon, the latter by a dbus/NetworkManager combo, or preferably a little hand scripting of the wpa_supplicant and iwlist programs. I leave this as an exercise to the reader, because if I have any extra time, I'll devote it to getting nosh or s6 running, or replacing the native init on CentOS or Fedora.

Adding the Udev and Network Services

# RUN simple shellscript to get udev running
# Udev is necessary to start network
# Run it before remounting / read-write.
# Don't bother to stop it at all
ObjectID=udevd
        ObjectDescription=Starting udev
        ObjectStartCommand=/root/udev_start.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=20
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN shellscript to set up network. Works on any distro having
#  the ip command. No config files necessary.
#  start it after Udev is up and / is read-write 
#  and the early Getty is running
ObjectID=network
        ObjectDescription=Setting up net devices and the network
        ObjectStartCommand=/root/upnet.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=50
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

Reboot, and verify that networking runs as expected.

Make Epoch Your Default Init

Up to this point, your Epoch Init System driven init has been under heavy construction, with a high probability of failure. For this reason, it was set up to init only if you added init=/e to the kernel line in Grub. But of course, time after time, tweaking the Grub kernel line becomes time consuming. So at this point, Epoch is probably stable enough on your system to make it your default. You've already copied your compiled epoch executable to /usr/bin/epoch, now copy it (owner/group root, executable by all) to /usr/bin/init or whatever file your kernel runs after setting itself up.

If you're having trouble determining what program the kernel runs as PID1, you're not alone. The ps command shows "init" regardless of the filename of the program run as PID1. Perhaps the best way of determining this is either looking in your Grub's config, or just trial and error, starting with /usr/bin/init or /sbin/int, depending on whether or not, respectively, your /sbin is just a symlink.

Add in /dev/mixer Capability

The sound card drivers are already loaded, probably by Udev. However, neither ALSA nor Pulseaudio is running, at last on my Manjaro box. And I like it that way! If your results are like mine, there is no /dev/mixer device. This makes volume control problematic. The step you're on now changes that.

Before anything else, use the package manager to install a text mode mixer such as rexima. Don't install Alsamixer because that depends on ALSA, and at this point you want to keep dependencies to a minimum. When you've installed your text mode sound mixer, run it. It will probably fail something like the following:

[root@openrc ~]# rexima
rexima: couldn't open mixer device.
[root@openrc ~]# ls -ldF /dev/mixer
ls: cannot access /dev/mixer: No such file or directory
[root@openrc ~]#

The preceding pretty much spells it out. You need /dev/mixer. And here's how you get it:

[root@openrc ~]# modprobe snd_mixer_oss
[root@openrc ~]# ls -ldF /dev/mixer
crw-rw---- 1 root audio 14, 0 29.12.2014 01:12 /dev/mixer
[root@openrc ~]#

When you run rexima again, it gives you the adjustments you expect. Now all that remains is to have this happen on boot. You want it to happen after the early Getty and after network instantiation, so if something goes wrong you can troubleshoot from the console or from ssh. And you want it to happen before any daemons. So add the following to your /etc/epoch/epoch.conf

# LOAD kernel module for /dev/mixer, without ALSA or PULSE
# Run after early Getty and network, but before daemons
# No need to stop it on shutdown.
ObjectID=devmixer
        ObjectDescription=Setting up /dev/mixer via oss
        ObjectStartCommand=/usr/bin/modprobe snd_mixer_oss
        ObjectStopCommand=NONE
        ObjectStartPriority=60
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

After adding the preceding to /etc/epoch/epoch.conf, reboot, and you should be able to run your sound mixer.

Add in Gettys

We already started an early Getty (tty2) to debug if things went south in the init. But typically, ttys 3 through 6 are also instantiated. Some systems put a getty on tty1, but I like to reserve tty1 for the boot messages. Your mileage may vary. I like to put these later gettys after the OS setup, but before the actual daemons. Gettys 3 through 6 don't need to start in order relative to each other. Their order relative to each other can be indeterminate, so their ObjectStartPriority can all be the same. For the purposes of this section, we make that priority 70: After OS setup, but before the daemons.

NOTE:

These later Gettys can actually be run just before, just after, or concurrent with the daemons. They have no dependencies, and nothing depends on them except that for them to log, they must be started after the filesystem becomes read-write. In this example, I make them happen just before the daemons.

Here are the services to add:

ObjectID=getty3
        ObjectDescription=Getty on /dev/tty3
        ObjectStartCommand=agetty tty3 &
        ObjectStopCommand=NONE
        ObjectStartPriority=70
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty4
        ObjectDescription=Getty on /dev/tty4
        ObjectStartCommand=agetty tty4 &
        ObjectStopCommand=NONE
        ObjectStartPriority=70
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty5
        ObjectDescription=Getty on /dev/tty5
        ObjectStartCommand=agetty tty5 &
        ObjectStopCommand=NONE
        ObjectStartPriority=70
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty6
        ObjectDescription=Getty on /dev/tty6
        ObjectStartCommand=agetty tty6 &
        ObjectStopCommand=NONE
        ObjectStartPriority=70
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

Reboot and you now have tty3, tty4, tty5 and tty6 added to the early tty2. If that's not the case, troubleshoot.

Add in Daemons

There's a certain class of programs commonly called daemons. They run in the background. If they abort, they're typically expected to restart automatically. After all, you don't want your web server to sit around busted until somebody notices it: Better it restart and add an entry to a log. Typical daemons are web servers, IMAP servers, SMTP mail servers, Samba servers, CUPS servers, time servers, database servers; the list goes on and on, although my advice is to only start the ones you really need.

These daemons have a lot in common. Most are servers listening for clients to interact with them. Most require the network, and therefore must be started after the network. And most of the time, they can be run in any order respective to each other, which for the Epoch Init System means they can usually all be given the same ObjectStartPriority.

These daemons should probably be among the first processes stopped during shutdown or reboot.

My guess would be that although most of these daemons could be started and stopped in any order relative to themselves, I would start any database servers before the other daemons, and I'd stop the database servers after all the other daemons are stopped. You want to protect the database as much as is humanly possible.

This section show makes an example of two daemons, sshd and crond. They're started after all the OS setup, and stopped before tearing down the OS for shutdown. Here are the two services as added to your /etc/epoch/epoch.conf:

# MANAGE sshd daemon
ObjectID=sshd
        ObjectDescription=Manage sshd daemon
        ObjectStartCommand=/usr/sbin/sshd
        ObjectStopCommand=PIDFILE /run/sshd.pid
        ObjectStartPriority=80
        ObjectStopPriority=10
        ObjectEnabled=true
        ObjectRunlevels=boot
        ObjectOptions=SERVICE AUTORESTART

# MANAGE crond daemon
ObjectID=crond
        ObjectDescription=Manage crond daemon
        ObjectStartCommand=/usr/bin/crond
        ObjectStopCommand=PIDFILE /run/crond.pid
        ObjectStartPriority=80
        ObjectStopPriority=10
        ObjectOptions=SERVICE AUTORESTART
        ObjectEnabled=true
        ObjectRunlevels=boot

Recap Your Results

If you've followed the procedure outlined in this major section (about getting Epoch to run), then your /etc/epoch/epoch.conf looks like this:

BootBannerText=Sorry, Lennart!
BootBannerColor=CYAN

Hostname=FILE /etc/hostname
DefaultRunlevel=boot
EnableLogging=true
DisableCAD=true
BlankLogOnBoot=true
MountVirtual=procfs sysfs devpts+ devshm+

ObjectID=getty2
        ObjectDescription=Early getty on /dev/tty2
        ObjectStartCommand=agetty tty2 &
        ObjectStopCommand=NONE
        ObjectStartPriority=40
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

# MAKE readwrite just after udev instantiates,
# MAKE readonly again after all killalls
ObjectID=rwfs
        ObjectDescription=root filesystem read-write support
        ObjectStartCommand=/usr/bin/mount -o remount,rw /
        ObjectStopCommand=/usr/bin/mount -o remount,ro /
        ObjectStartPriority=30
        ObjectStopPriority=100
        ObjectEnabled=true
        ObjectRunlevels=boot 

# KINDLY request all processes to end themselves
# Give them 2 seconds to do so.
ObjectID=killall5_soft
        ObjectDescription=Terminating all processes
        ObjectStopCommand=killall5 -15 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=90
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

# MURDER all remaining processes
# Give them 2 seconds to do so.  
# Then hand control to service rwfs to 
# remount / read-only to prevent filesystem corruption     
ObjectID=killall5
        ObjectDescription=Killing all processes
        ObjectStopCommand=killall5 -9 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=91
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

ObjectID=sysclock
        ObjectDescription=Configuring system clock
        ObjectStartCommand=hwclock -s
        ObjectStopCommand=hwclock -w
        ObjectStartPriority=35
        ObjectStopPriority=80
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN simple shellscript to get udev running
# Udev is necessary to start network
# Run it before remounting / read-write.
# Don't bother to stop it at all
ObjectID=udevd
        ObjectDescription=Starting udev
        ObjectStartCommand=/root/udev_start.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=20
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN shellscript to set up network. Works on any distro having
#  the ip command. No config files necessary.
#  start it after Udev is up and / is read-write 
#  and the early Getty is running
ObjectID=network
        ObjectDescription=Setting up net devices and the network
        ObjectStartCommand=/root/upnet.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=50
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# LOAD kernel module for /dev/mixer, without ALSA or PULSE
# Run after early Getty and network, but before daemons
# No need to stop it on shutdown.
ObjectID=devmixer
        ObjectDescription=Setting up /dev/mixer via oss
        ObjectStartCommand=/usr/bin/modprobe snd_mixer_oss
        ObjectStopCommand=NONE
        ObjectStartPriority=60
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

ObjectID=getty3
        ObjectDescription=Getty on /dev/tty3
        ObjectStartCommand=agetty tty3 &
        ObjectStopCommand=NONE
        ObjectStartPriority=70
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty4
        ObjectDescription=Getty on /dev/tty4
        ObjectStartCommand=agetty tty4 &
        ObjectStopCommand=NONE
        ObjectStartPriority=70
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty5
        ObjectDescription=Getty on /dev/tty5
        ObjectStartCommand=agetty tty5 &
        ObjectStopCommand=NONE
        ObjectStartPriority=70
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty6
        ObjectDescription=Getty on /dev/tty6
        ObjectStartCommand=agetty tty6 &
        ObjectStopCommand=NONE
        ObjectStartPriority=70
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

# MANAGE sshd daemon
ObjectID=sshd
        ObjectDescription=Manage sshd daemon
        ObjectStartCommand=/usr/sbin/sshd
        ObjectStopCommand=PIDFILE /run/sshd.pid
        ObjectStartPriority=80
        ObjectStopPriority=10
        ObjectEnabled=true
        ObjectRunlevels=boot
        ObjectOptions=SERVICE AUTORESTART

# MANAGE crond daemon
ObjectID=crond
        ObjectDescription=Manage crond daemon
        ObjectStartCommand=/usr/bin/crond
        ObjectStopCommand=PIDFILE /run/crond.pid
        ObjectStartPriority=80
        ObjectStopPriority=10
        ObjectOptions=SERVICE AUTORESTART
        ObjectEnabled=true
        ObjectRunlevels=boot

To demonstrate the simplicity of this setup, I sshed into the experimental computer from my daily driver desktop, then on the experimental computer, I logged into tty4 as user slitt, used dmenu to run Openbox, and ran xterm in the Openbox session. Then I performed a pstree -A command from the ssh session, and it looked like this:

init-+-4*[agetty]
     |-at-spi-bus-laun-+-dbus-daemon
     |                 |-{dconf worker}
     |                 |-{gdbus}
     |                 `-{gmain}
     |-at-spi2-registr---{gdbus}
     |-bash---gnumeric---sudo
     |-bash---xterm---bash
     |-crond
     |-dbus-daemon
     |-dbus-launch
     |-gconfd-2
     |-login---bash---startx---xinit-+-Xorg.bin---{Xorg.bin}
     |                               `-openbox
     |-lxpolkit
     |-manjaro-setting---{QXcbEventReader}
     |-pamac-tray---{gdbus}
     |-pnmixer
     |-sshd---sshd---sshd---bash---su---bash---pstree
     |-udevd
     |-xfce4-clipman---{gmain}
     `-xfconfd

But wait, there's more. Watch what happens when I close xterm and log out of Openbox, so no X is running:

init-+-4*[agetty]
     |-crond
     |-login---bash
     |-sshd---sshd---sshd---bash---su---bash---pstree
     `-udevd

Now that's a level of simplicity a guy could get used to!

Tell me one more time just so I'll understand: How would systemd benefit me?

Convert priority numbers to constants

If you've been following along, you've seen priority numbers get more and more confusing. Even though you know the process dependencies and order, you need to search the entire /etc/epoch/epoch.conf to rearrange order or even fathom order. It doesn't have to be this way.

In the top, global part of /etc/epoch/epoch.conf, you can define a bunch of priority constants with the DefinePriority keyword. For instance, to set constant Typical_Daemon_Start to 80, you'd place the following line in the global section:

DefinePriority=Typical_Daemon_Start 80

Now, within a specific service, you can place the following line:

ObjectStartPriority=Typical_Daemon_Start

The preceding sets the service's ObjectStartPriority to 80, but that can be changed in the header. Because you group all the DefinePriority statements one after another, it's very easy to correctly order or reorder your services. The following is what your /etc/epoch/epoch.conf looks like after replacing the numbers with same-numbered constants:

BootBannerText=Sorry, Lennart!
BootBannerColor=CYAN

Hostname=FILE /etc/hostname
DefaultRunlevel=boot
EnableLogging=true
DisableCAD=true
BlankLogOnBoot=true
MountVirtual=procfs sysfs devpts+ devshm+

# DEFINE PRIORITY CONSTANTS
DefinePriority=Rwfs_Start 30
DefinePriority=Early_Getty_Start 40
DefinePriority=Sysclock_Start 35
DefinePriority=Udev_Start 20
DefinePriority=Network_Start 50
DefinePriority=Dev_Mixer_Start 60
DefinePriority=Typical_Daemon_Start 80
DefinePriority=Typical_Getty_Start 70
DefinePriority=Display_Manager_Start 120


DefinePriority=Display_Manager_Stop 40
DefinePriority=Typical_Daemon_Stop 10
DefinePriority=Sysclock_Stop 80
DefinePriority=Killall5_Soft_Stop 90
DefinePriority=Killall5_Stop 91
DefinePriority=Rwfs_Stop 100 



ObjectID=getty2
        ObjectDescription=Early getty on /dev/tty2
        ObjectStartCommand=agetty tty2 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Early_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

# MAKE readwrite just after udev instantiates,
# MAKE readonly again after all killalls
ObjectID=rwfs
        ObjectDescription=root filesystem read-write support
        ObjectStartCommand=/usr/bin/mount -o remount,rw /
        ObjectStopCommand=/usr/bin/mount -o remount,ro /
        ObjectStartPriority=Rwfs_Start
        ObjectStopPriority=Rwfs_Stop
        ObjectEnabled=true
        ObjectRunlevels=boot 

# KINDLY request all processes to end themselves
# Give them 2 seconds to do so.
ObjectID=killall5_soft
        ObjectDescription=Terminating all processes
        ObjectStopCommand=killall5 -15 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=Killall5_Soft_Stop
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

# MURDER all remaining processes
# Give them 2 seconds to do so.  
# Then hand control to service rwfs to 
# remount / read-only to prevent filesystem corruption     
ObjectID=killall5
        ObjectDescription=Killing all processes
        ObjectStopCommand=killall5 -9 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=Killall5_Stop
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

ObjectID=sysclock
        ObjectDescription=Configuring system clock
        ObjectStartCommand=hwclock -s
        ObjectStopCommand=hwclock -w
        ObjectStartPriority=Sysclock_Start
        ObjectStopPriority=Sysclock_Stop
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN simple shellscript to get udev running
# Udev is necessary to start network
# Run it before remounting / read-write.
# Don't bother to stop it at all
ObjectID=udevd
        ObjectDescription=Starting udev
        ObjectStartCommand=/root/udev_start.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=Udev_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN shellscript to set up network. Works on any distro having
#  the ip command. No config files necessary.
#  start it after Udev is up and / is read-write 
#  and the early Getty is running
ObjectID=network
        ObjectDescription=Setting up net devices and the network
        ObjectStartCommand=/root/upnet.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=Network_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# LOAD kernel module for /dev/mixer, without ALSA or PULSE
# Run after early Getty and network, but before daemons
# No need to stop it on shutdown.
ObjectID=devmixer
        ObjectDescription=Setting up /dev/mixer via oss
        ObjectStartCommand=/usr/bin/modprobe snd_mixer_oss
        ObjectStopCommand=NONE
        ObjectStartPriority=Dev_Mixer_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

ObjectID=getty3
        ObjectDescription=Getty on /dev/tty3
        ObjectStartCommand=agetty tty3 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty4
        ObjectDescription=Getty on /dev/tty4
        ObjectStartCommand=agetty tty4 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty5
        ObjectDescription=Getty on /dev/tty5
        ObjectStartCommand=agetty tty5 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty6
        ObjectDescription=Getty on /dev/tty6
        ObjectStartCommand=agetty tty6 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

# MANAGE sshd daemon
ObjectID=sshd
        ObjectDescription=Manage sshd daemon
        ObjectStartCommand=/usr/sbin/sshd
        ObjectStopCommand=PIDFILE /run/sshd.pid
        ObjectStartPriority=Typical_Daemon_Start
        ObjectStopPriority=Typical_Daemon_Stop
        ObjectEnabled=true
        ObjectRunlevels=boot
        ObjectOptions=SERVICE AUTORESTART

# MANAGE crond daemon
ObjectID=crond
        ObjectDescription=Manage crond daemon
        ObjectStartCommand=/usr/bin/crond
        ObjectStopCommand=PIDFILE /run/crond.pid
        ObjectStartPriority=Typical_Daemon_Start
        ObjectStopPriority=Typical_Daemon_Stop
        ObjectOptions=SERVICE AUTORESTART
        ObjectEnabled=true
        ObjectRunlevels=boot

Optional: Add in Display Manager

How important is a display manager?

Personally, I'm a startx kinda guy. I've had far too many computers have X hang the whole box for me to comfortably boot to GUI. Ctrl+Alt+F2, then username, password, and startx aren't a huge imposition on me. They're not difficult to teach others: I taught my kids to log into their Linux laptops that way.

My Manjaro/Epoch display manager isn't "ready for prime time", but if one insists on booting to GUI, it works. First, here's the new service to add to your /etc/epoch/epoch.conf:

# RUN ONCE Display Manager.
# Runlevels=boot to enable, Runlevels=gui to disable to GUI via startx
# NOTE: Display Manager might invoke dbus.
# NOTE: Toggle ObjectEnabled to turn GUI booting on or off.
ObjectID=DisplayManager
        ObjectDescription=DisplayManager
        ObjectPIDFile=/run/DisplayManager.pid
        ObjectStartCommand=/usr/bin/lxdm &
        ObjectStopCommand=PID
        ObjectStartPriority=Display_Manager_Start
        ObjectStopPriority=Display_Manager_Stop
        ObjectEnabled=true
        ObjectRunlevels=boot

You're not nearly done. For one thing, you need to enable reboot and shutdown from your display manager and window manager. I got it to work using some shellscripts. Your mileage may vary.

Before making any of these shellscripts, back up the current ones, which were contributed by your former Init System. You can always put them back if you don't end up liking Epoch.

Create the following /usr/bin/reboot:

#!/bin/sh
xterm -fn 12x24 -e 'echo "Type your password to reboot, Ctrl+C to abort" & sudo /usr/bin/epoch reboot'
echo "Type your password to reboot, Ctrl+C to abort" 
sudo /usr/bin/epoch reboot

Copy that same file to /usr/bin/restart. Next, create the following /usr/bin/poweroff:

#!/bin/sh
xterm -fn 12x24 \
-e 'echo "Type your password to halt and power down, Ctrl+C to abort" & sudo /usr/bin/epoch poweroff'
echo "Type your password to halt and power down, Ctrl+C to abort" 
sudo /usr/bin/epoch poweroff
[root@openrc ~]#

Copy that same file to /usr/bin/shutdown. Obviously, all four scripts should be executable by all, and all four should be owner root, group root. Those four files enable the lxdm display manager to reboot or shut down via its normal GUI interface. I can't speak to other display managers, nor can I state what might happen on other distros.

Next, use LXDE to create the following normal user Desktop Shortcut for rebooting from LXDE:

General screen of reboot shortcut Desktop Properties screen of reboot shortcut

And create the following normal user Desktop Shortcut for powering down from LXDE:

General screen of shutdown shortcut Desktop Properties screen of shutdown shortcut

As far as logging out of LXDE, the normal icon should bring up a window allowing you to log out of LXDE back into the display manager.

Note that if you're using Openbox, the best way to do it is to use dmenu to run the /usr/bin/reboot or /usr/bin/poweroff you created earlier, and the best way to log out of Openbox is to use the mouse or a keyboard shortcut to get Openbox's root menu, then choose "log out" or "quit" or whatever it's called on your installation.

Don't Expect Miracles

If you're looking for that FreeDesktop.org, Kumbaya, we're all talking to each other including the init and the window manager feeling, you'll likely be disappointed with this setup. You'd need to do a lot more work to get an Epoch initted system to deliver most of the unique features of Unity, Gnome or KDE. But then, if you're using Unity, Gnome, or KDE, there's a good chance you'll prefer systemd anyway.

The preceding being said, if you want a well running desktop that boots Grub to lxdm in eight seconds on an 8 year old computer, and are willing to forego just a few bells and whistles, this will work very well for you.

NOTE:

This section, Getting Epoch Running, was completed and published to the Internet on 12/30/2014.

Editorial: Is That All There Is?

So it's the afternoon of 12/29/2014. I'm putting the finishing touches on my Epoch Init System content. Between Epoch's simplicity and its excellent documentation, replacing your current init system with Epoch is a 2 hour job, no package manager necessary. And, using my runit documentation, replacing your current init with runit is probably a five hour job. Which brings up the obvious question:

Is that all there is?

The four month war on the Debian-User mailing list. The back and forth between press pundits. The alleged contract on Poettering's life and subsequent "Open Source is quite a sick place to be in" whine by Poettering. The shouts of "FUD" on LUG mailing lists and IRC channels every time someone discussed init system choice. All of this was solvable with less than a day's work by computer literate Linux users. When I finally found the solution, all I could think was:

Is that all there is?

Obviously, I need to take some responsibility for going over the top during the Debian-User systemd wars. Obviously, my fellow anti-systemd people need to take some responsibility too. However, I assign the lion's share of responsibility to the pro-systemd forces. Here's why...

Back in the spring, summer and fall of 2014, "just plain users" had little understanding of init systems. Why should they understand: Sysvinit had been a constant for ten years, sometimes augmented by Upstart, neither of which was especially controversial. To users, the init system was a black box that pretty much "just worked."

Now contemplate those who knew all about init systems:

If, before the 7/1/2014 escalation of the Debian-User systemd war, somebody from any of the preceding four entities had taken a couple days to write something like the document you're reading right now, the whole conflict would have been avoided. It took me over a month to do the research and experimentation to put this document together: Those guys had the init system knowledge to write this document pretty much in real time. Not only that, but a lot of those guys were paid for their Open Source work.

Such a document, written by such a paid, knowledgeable person, before July 1, 2014, would have given users an easy alternative to systemd on any distro. Most anti-systemd sentiment would have dissipated as users simply bolted a runit or Epoch init to their operating system, and went merrily on their way. If just one person from the "systemd cabal", Red Hat, or the Debian Developers, had written and publicized a page like this one in a timely fashion, they could have prevented the whole conflict.

But that's not their style.

They knew every technical fact on this page. They had the time and the money and the manpower to create and publicize this page, heck, a better page, in a timely manner, thus avoiding the four month conflict. They chose not to.

Instead, they spewed FUD contrary to the facts. They said the only init systems "ready for prime time" were systemd, Upstart and sysvinit, implying that the likes of runit, Epoch, s6, nosh, and OpenRC were rickety junk. That's nonsense. They said we needed parallel startup of services, so boot didn't take five minutes. For the vast majority of us, server and desktop alike, who were not booting up fifty services, that's nonsense. They flat out implied that were three classes of init systems: systemd, Upstart, and all the rest, as if runit, s6, nosh and Epoch have anything in common with the hopelessly opaque, inefficent and antiquated sysvinit. They said we needed more reliable init systems. How you can get more reliable than Epoch, or runit with very simple service dependency aware start scripts, is beyond me. Why the FUD? Why the FUD when the obvious impending result was civil war?

Why indeed? Think about it. What did they not want us to know?

Think about it long and hard. They knew the truth; they chose not to share it. In fact, they spread antitruths. You don't have to voice your opinion out loud: Nobody likes to be called a conspiracy theorist. Just think about it. What was their motive?

We're three days from 2015. The systemd wars are over, probably to everyone's delight. Like many wars, there's no clear winner. Red Hat succeeded in co-opting almost every distro, but spawned groups of technologists ready, willing and able to undo Red Hat's "technology". And Red Hat suffered serious damage to their brand, at least among many Linux users willing and able to use the command prompt and editors to do some light configuration, scripting, and maybe some coding.

Debian succeeded in making systemd the default init on their new Jessie version, and their "support" for other init systems is still being debated. But they lost developers, and they lost some great user talent, and I mean people who could code. Great talent lost to alternate distros, alternate inits, and even alternate operating systems.

Those of us not wanting systemd on our boxes obviously lost the war. We either became refugees living in FreeBSD and Gentoo land, or are busy building our own systemd-less Debian, or are creating ways to init with systems declared "not ready for prime time", but still using Red Hat's pet Udev, and soon to be using Red Hat's pet Wayland (we need to start gearing up for that battle right away).

But before the forces of systemd celebrate the vanquishment of the anti-systemd rebels, they should consider that we rebels are battle hardened, and we know far more about system software than 90% of the Linux populace knew pre-systemd. While they've been learning talking points, we've been learning tech. We're coding software and making documentation to make systemd optional. And we're a tad bit upset with the powers that be. We're strong, smart and motivated.

Concerning Devuan, #debianfork, the modular-debian mailing list, and the people currently reproducing the Manjaro Experiments, the systemd patriots have this to say:

Is that all there is?

It's exactly that kind of myopia that will lead to their downfall.

inotifywait -m -e CREATE,DELETE /dev/usb

So at about 3am the morning of December 31, 2014, a guy on the #debianfork IRC mentioned that the Debian Tech guys had limited the init selection to event driven inits.

Oops! I had forgotten that systemd and Upstart were event driven, rather than just good at handling process dependencies. I had to step back and think for a moment. And then I asked my usual question, "why?" I asked him why the Debian Tech guys felt event driven inits were necessary. The guy replied something to the effect "you want to be able to respond when someone plugs in a drive or the network changes."

Yeah, that's not just propaganda, it's the truth. Hoping there could be an alternative to ruling out good inits like runit and Epoch, I thought for a couple minutes. I understood the need, but is init the right place to put such event detection? Disgruntled, I just said that I'd rather poll dmesg every second than run an overly complex init. A couple guys agreed with me, and one of them stated he does a lot of polling.

Yes, polling is good enough for Steve Litt, and a minor polled operation once per second consumes negligible resources. But recommending polling is a tremendous propaganda victory for "the other side." I sat and thought of how such things could be interrupt driven.

And within a second, I remembered that Linux (and the Linux kernel alone) has a library that detects disk changes as interrupts. I had forgotten the name, had to websearch it. It's the inotify library. Twenty minutes I had this command watching my computer's USB system:

inotifywait -m -e CREATE,DELETE /dev/usb

When you run the preceding command, it prints /dev/usb/ CREATE hiddev1 every time I plugged in my USB mouse, and /dev/usb/ DELETE hiddev1 every time I unplugged it. The rest of the story doesn't take a lot of imagination...

I could pipe the inotifywait command into a program that, every time it gets "CREATE" on stdin, runs dmesg, parses its most recent output, and deduces the partitions on the newly plugged device. It can then write the device's information into a fifo going into a service which does the mounts.

Am I going to actually try to make this work? Doubtful, I don't need it at this point. Am I aware that I proved the concept of just one use case? Of course I am, but I did it in less than an hour, and I see no reason why this concept couldn't be used for anything one wants to be event driven. Do I agree it's a kludge because I'm duct taping together all these different processes instead of writing C code? That depends on which of the four definitions of "kludge" one subscribes to, but that doesn't matter: There's an inotify library, and most of what I just discussed could be done in C by a sufficiently motivated person who has the time.

I look at this whole thing, and see the following process:

  1. FreeDesktop.Org, Redhat, or the systemd cabal put yet another complication into init.
  2. Folks valuing simplicity object to the new addition.
  3. The RedHatians say it's needed in order to achieve such and such.
  4. A simplicity valuing person easily makes a solution to achieve such and such outside of init (and probably outside the kernel space too).
  5. The Redhattians respond that the simple solution is just [insert negative characterization here].
  6. Systemd fanboys parrot the negative characterization.
  7. If the simple solution is documented halfway decently, the simplicity lovers start using it, and let others know that it works.
  8. Lather, rinse, repeat.

You know what I think? I think the preceding process, iterated over time, will be the undoing of systemd and lead to a Renaissance in public participation in the coding, configuration and architecture of Linux. Because, in almost every case, there are many ways to accomplish something, and the kernel or init system is usually the wrong place to accomplish it.

So, one day when your curiosity is bigger than your workload, try the following command, plug and unplug USB devices, and contemplate the possibilities:

inotifywait -m -e CREATE,DELETE /dev/usb

The Four Definitions of Kludge

It's 8pm on 12/31/2014, and it's a little in doubt whether I'll finish this section before my wife and I go to a New Years party. But let me start, at least.

When somebody calls my solution a "kludge", I think that characterization means one of four things:

  1. I fixed a symptom instead of the root cause.
  2. I built something so rickety that a moderate use case change or environment change will break it.
  3. My solution is functional but ugly.
  4. My solution removes the justification for the software they wrote or are writing.

After I determine which of the four applies (and they're not necessarily mutually exclusive), I respond as follows:

  1. I get in there and fix the root cause.
  2. I find other software that does the job, or find ways to make my solution work without all sorts of nested ifs and other complications.
  3. I know it's a matter of aesthetics, so I ask myself whether prettier would be worth the extra cost, and act accordingly.
  4. I proudly wear the word "kludge" as a badge of honor.

Not all "kludges" are created equal.

Happy New Year

It's 2:10am on 1/1/2015. My wife and I just got back from our New Years Party. It was a great party, and everyone had fun.

Now I'd like to propose a toast to GNU/Linux users worldwide. Let's all raise our glasses and toast 2015: The year of simplicity and choice.

Happy New Year!

Litt Repelled At the First Battle of Red Hat

If you've been reading what I've been writing, you knew that sooner or later I'd take the fight to Red Hat. That war's first skirmish happened on 1/1/2015, when I kinda-sorta initted CentOS 7.0 with the Epoch Init System. Bottom line: My attack was repelled, but I did manage to gain some ground.

When you try to put an alternate init onto CentOS (the closest you can get to RHEL without payments and service contracts), you know you're in hostile territory. I felt like Alexander the Great in his Indian Campaign, Red Hat/Gnome's poor X performance was the Ganges River, and various Redhatisms were the enemy's war elephants. Progress was slow and treacherous. Here's how it happened...

In the late afternoon or early evening I installed CentOS 7.0 on the same experimental computer that had housed evaluations of Porteus, FreeBSD, OpenBSD, PC-BSD, PC-LinuxOS and Manjaro. The only hardware difference was the hard disk. At first I had done the minimal install, but couldn't figure out how to install X and LXDE. So, to save time, I did the Gnome Install. Because I was wire-jumpering an alternate init around the native init, both GUI web browsing and cut and paste were vital to configuration, experimentation and troubleshooting.

The Gnome Install fought me tooth and nail. The systemd init booted from Grub to X in 10 seconds: A little more than Epoch and a little less than runit. I know it was in X because the screen got a lighter black. But then it took about 30 seconds for that lighter black screen to display the gdm login screen, and a significant time after login to stabilize into Gnome.

Worse still, every time I did anything with the GUI, the GUI took 20 seconds to recover from what I did. I once managed to drag a terminal, that drag blacked out everything it had travelled over, and it took at least 20 seconds to return to a useful interface. You can't work that way.

NOTE:

I suspect this incredibly degraded performance was due to the disk I was using, as another disk loaded with CentOS non-LVM with ext4 instead of xfs filesystems, performed within reasonable expectations. A complete answer won't be available until I lay down a default CentOS install on a known good disk.

No problem, I'd just install LXDE. Oops, yum search found no LXDE. It found no Xfce. I put out the call on various mailing lists and IRC channels: How do I install LXDE on CentOS 7.0? Nobody knew. I kid you not, nobody knew. Finally, almost a day later, somebody told me of this set of commands:

yum install epel-release
yum search xfce

LXDE was just plain absent from "EPEL", as well as the default CentOS package groups. In addition, there didn't appear to be a meta package for all of Xfce: I had to install several packages. And when I was done, Xfce was not useful. You see, Xfce, Red Hat style, has apparently taken a page from Unity's book and displayed the browser and/or terminal at the upper left corner, with its titlebar and menu covered by the Gnome taskbar. The Alt+left drag that moves stuck windows in other environments didn't work here. I couldn't change the terminal's or the browser's tiny fonts, so work within the GUI was once again impossible.

NOTE:

When I later installed CentOS, without LVM, with ext4 instead of xfs, on a known good hard disk, Xfce was useable and had the right geometry.

So I just started working with the ttys. However, the ttys were set to the tiniest imaginable font, so work was slow. And I couldn't copy and paste, probably because I hadn't yet put a GPM service in Epoch yet. Work was soooo slooooo!

Then I installed the lynx and links browsers, did a text dump of this web page to Vim cut and paste into /etc/epoch/epoch.conf, as well as the various shellscripts in /root I use to start up udev, as well as the shellscript I use to bring up and configure the wired Ethernet port. Slowly I fixed one problem after another. There was an issue with my keyboard typing "m" instead of "n", which was exacerbated by the teeny tiny tty font so that I couldn't see the difference, but I finally got all that fixed within an hour.

Oh, and there are the Grub tweaks necessary to get it to fire up Epoch at all. I had copied the epoch executable to /e for simple Grub line editing, but even so, it wasn't until I eliminated the phrase "rhgb" from the line that it ran /e instead of just ignoring it and booting Gnome anyway.

So, somewhere around 10pm I had CentOS initting with Epoch instead of systemd, complete with full networking capability. However, it had nothing in the /home directory, and I'm pretty sure it had no swap partition either. To the best of my understanding, the lack of swap and /home is an LVM problem, and I know absolutely nothing about LVM. In addition, when I ran startx as user root, Xfce fired up instantly, but without mouse support.

I tried for an hour to push through the LVM problems, but in the end decided that was a bad use of my time, and am later going to follow Epoch Init System author Subsentient's advice and reinstall CentOS without LVM. After I get that working perfectly, I'll Rapid Learn LVM and, armed with knowledge, Epochify a default CentOS 7 box.

Progress

My assault was repelled, but I still gained some territory. I can install Xfce. I can boot to a root-only Epoch init, complete with networking and the ability to run the lynx links CLI web browsers (good enough to do downloads), and the weechat IRC client, to get some help. From the viewpoint of a functional desktop, these pieces of progress aren't significant. But from a strategic point of view, I've set myself up to make my next assault straight for the heart of a no LVM, pure Epoch initted CentOS.

1/7/2015: My First Init Presentation

So yesterday, 1/7/2015, I gave my first Init Presentation at Greater Orlando Linux User Group (GoLUG). Here are the slides:

http://www.troubleshooters.com/linux/presentations/golug_inits/golug_inits.pdf

I demonstrated Manjaro 8.11, the default systemd install, with both Epoch and runit jumpered around systemd. So if I booted it with no Grub changes, it booted with a systemd init. If I added init=/r, it used runit for init. If I booted with init=/e, it used Epoch.

Lessons Learned

It went well, but I think it could have done better. So, in case you'll later be presenting on Init Systems, here are some lessons I learned...

Projectors Are A Challenge

Even in the best of times, using a projector or huge TV with a Linux laptop is a challenge. When you've been messing with your Init System, it doesn't get any easier, and may get harder. The next time I give this presentation, if it's anywhere I drive to, I might bring a desktop computer. Naturally, I'll need to be sure that, one way or another, I have HDMI as well as SVGA coming out of that desktop, to accommodate whatever they have. As a fallback, I might also bring my own 28" monitor. Better than nothing, and somewhat useable at proper font sizes for a crowd up to 10 people.

As always, lxrander was extremely helpful. In fact, I got the giant TV to show my video, but the giant TV insisted on printing "wrong resolution" error messages on top of my perfectly-rendered video. So I had everyone view the slides from the Internet, but not everyone had a computer, and the Firefox PDF renderer doesn't do whole pages, and I was working off a paper copy, well, you get the picture.

Have a Plan B

If your experience is anything like mine, Linux' use of projectors are iffy in the best of situations, and probably more so in an alternate init situation. Every different projector or huge TV presents a different set of challenges. My finding is that, often, I can see virtual terminals but not X on the projector, or vice versa. My new HP laptop ignores the fn+F4 key combo that supposedly switches between external, internal, and both. The only way I've ever found to show both in Linux was to put them both on the exact same resolution, and that typically doesn't work well.

With all this going on, if you can't bring your own known-good and known-config projector, I'd have screenshots of all relevant bootup and virtual terminal output, and, if your presentation is short and there aren't too many attendees, paper copies of your presentation, so if the projector or TV just can't rise to the occasion, your presentation can go on unmolested.

Watch Out For Wifi

If you do DIY init's like I do DIY inits, you're not running dbus, nor NetworkManager or wicd, nor a window manager with a panel to even view NetworkManager with, so you're probably using a combination of wpa_supplicant and either dhclient or dhcpcd. These types of "Roll Your Own" wifi access methods are trickier than just clicking the NetworkManager icon, choosing an SSID, and entering a password.

So you want to get your wifi-fu down to a science long before you step into the classroom. Make sure your computer can log in to public Wifi, WPA and WPA2 Wifi, Wifi whose SSIDs have spaces and punctuation, everything. You need to become a wpa_supplicant ninja. Programs wpa_supplicant, dhclient, dhcpd and pump are scantily documented, complex, and often indeterminate, so this won't be easy, especially if you're mobile and connect to a wide variety of Wifi transmitters.

The wicd Option

The alternative is to become a dbus ninja and get wicd running. This isn't always easy, so it's a "pick your poison" type of situation. Solutions requiring dbus are often easier to use, but have dependency entanglements on dbus and other things, and they can also be indeterminate. Dbus can be a real mess to get running correctly, especially on an ex-systemd machine.

Still, my suggestion would be to start with a combination of dbus and wicd, with its wicd server, as well as its wicd-curses and wicd-gtk clients. My experience is that you're much more likely to succeed with wicd-gtk than with wicd-curses.

The first thing you need to do is get dbus running. Systemd machines get dbus running in a very, very special way, that requires systemd. If you jumper around systemd, you don't want to instantiate your dbus with all sorts of systemdisms. Instead, use the following command, which seems to work in a wide variety of situations:

mkdir -p /var/run/dbus && dbus-uuidgen --ensure && dbus-daemon --system

The preceding is probably enough for wicd-curses, because it seems like running X does more stuff to dbus, and that stuff is necessary to authenticate with wicd. So I'd recommend, for an easier life, run X, and then run wicd-gtk, and log in.

But even that's not enough. If your life is anything like mine, you'd better be sure to enable debug mode. Debug mode is in the "Advanced Settings" of Preferences. Once you turn on debug mode, you'll see message phrases similar to "bringing up wlo1", then "authenticating", then "getting IP address" at the bottom of the wicd-curses screen, or somewhere on the wicd-gtk screen if you're uing that.

If the client gets stuck at "authenticating" and you're using wicd-curses, run X and try wicd-gtk. If it gets stuck at "getting IP address", try changing your "DHCP Client" (available via preferences->External_programs). At a no-security McDonalds Wifi, I had a reproducible situation in which I couldn't get an IP address with this setting at "Automatic", but could get one with it set at "dhclient".

Prepare Hugely

Be sure to try your computer on an external monitor to find out how it works under those circumstances. Maybe try more than one external monitors.

Take your computer to a couple Wifi hotspots to make sure your chosen init's wifi config works correctly under a wide variety of circumstances. Wifi is twitch beyond belief, so you should walk in knowing you've mastered it.

Convert your presentation to PDF, and use the evince program to control it. Make sure that the "presentation mode" of evince, usually invoked with F5, works correctly, such that with an external wireless mouse, left click advances one slide, right click goes back one slide, and the mouse wheel can quickly advance and go back through several slides.

Getting your alt-init laptop to correctly operate a projector is a big credibility builder. Being forced to use the systemd config to show your slides makes your point less credible. Having to use a Windows box to present makes it even less credible. But the biggest credibility killer, by far, is if you try and fail to show your presentation with your alternate init, because such a failure is a graphic demonstration of your alternate init's impracticality. Whatever you do, don't fail.

Set up early

Like I said, failing to operate your projector with your alt-init is the kiss of death. Likewise, failure to get the network makes your message sound like factless hype. So get there early. Like an hour early if possible. Set up, make everything work, reboot, and make sure it still works. If setting up takes 5 minutes, congrats, you have 55 minutes to go over what you're going to say. But if it takes an hour, you still start your presentation with a perfectly working alt-init, presenting a totally believable message.

If you absolutely can't get your alternate init to operate the Wifi or Projector, present it booted to systemd, make no secret of that fact, and mention that your alternate init works well at your place, but has a couple bugs you still need to shake out.

If you have your own projector, bring it!

Imagine the luxury of learning exactly how your computer will interact with the projector, in the privacy of your own office, with nobody to see all the experimentation leading to the perfect presentation. That's what life is like if you bring your own projector.

Just be sure that the venue has a small, moveable table on which you can sit your projector. Many projectors work only a certain distance from the screen, and if you can't put a projector near that spot, your presentation has a problem.

Outcome of my first alternate-init presentation

If I'd brought printouts, everything would have been just fine. I could have shown everybody my originally-systemd computer that now inits with either Epoch or runit, and then run the actual presentation from paper. Instead, I needed to have those with laptops view the PDF in Firefox's PDF viewer, with those without a computer having to crowd around those who had one.

If you are forced to have people view your PDF on Firefox, depending on your Firefox version, you might have a "presentation mode" icon in the PDF display area. If you have it, it looks like a video's "fullscreen" icon. With Firefox Presentation Mode, slides display a page at a time, exactly on page boundaries, with the page covering the full screen. This is good.

Better still, right clicking your wireless mouse advances one slide. If you need to go back a slide, use the mouse's scroll wheel, because right click just brings up a menu. I've found that Logitech M310 mice are great for presentations, because their scroll wheels have significant back pressure between ratchets, so they're easy to back up one slide at a time, and they don't accidentally go forward or back as easily as some of the newer "fast scroll" mice.

Bottom line: My presentation was certainly in the bottom half of all my presentations, but it wasn't a disaster. Next time I'll do better.

The Cost of Dependencies

On the face of it, things like NetworkManager and Wicd are great, considering that the alternative appears to be searching what's available with iwlist wlo1 scanning, then editing wpa_supplicant.conf and inserting several lines that appear to be a poorly documented magical incantation. Yes indeed, as long as you're using your computer the way the distro designers envisioned, NetworkManager and Wicd are heaven sent.

The problem comes when you go a little offroad, like try to lighten the load by removing dbus, and maybe substituting another init for systemd. Or, with NetworkManager, doing something as simple as using a window manager lacking a taskbar and tray. So if you want NetworkManager or Wicd, you need to go back and restore dbus, and perhaps run a panel. Now take dbus, how should you run it? Perhaps this:

/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
/usr/bin/dbus-launch --autolaunch 9533c95cde1e4222a1ea6b92730eb8c4 --binary-syntax --close-stderr
/usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
/usr/lib/at-spi2-core/at-spi-bus-launcher
/usr/bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
Or maybe this: 
/usr/bin/dbus-daemon --system
/usr/bin/dbus-launch --autolaunch 9533c95cde1e4222a1ea6b92730eb8c4 --binary-syntax --close-stderr
/usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
/usr/lib/at-spi2-core/at-spi-bus-launcher
/usr/bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3

Or this something else? What the heck do the preceding even mean?

Dependency Chains

MyWonderApp requires LibA and LibB. No problem. Except LibA requires LibWeirdX and LibWeirdY, and LibB requires LibWeird1 and LibWeird2. All of the LibWeird libraries depend on several different LibObscure libraries, which in turn require several LibBetchDidntKnowThisExists libraries. All the libraries must be newer than some version number (different versions in all cases), but there's no package for some of the new ones, and besides, one of the distant descendants requires a newer glibc, and several of the descendants just won't compile at all.

Hyper-dependent programs are no problem when they're included in the distro's package manager, and when the package manager's version is both bug-free and modern-featured. But if the program isn't in the package manager repositories, or the distro's version of the program is buggy (which happens quite often), or when you really need that one new feature that a later version has but your distro's version doesn't, or when your distro compiled in the wrong features, you need to compile from scratch. And when you do, dependency chains are going to make your life a living hell.

And, of course, if there's a memory leak or other bug in any of the program's dependencies, then there's a memory leak or other bug in the program. I'd be a rich man if I got a dime every time I heard a program's author say "there are no bugs in my code, it's a bug in LibKitchenSink", as if that makes buggy software any easier to use.

The more you need the program, the more hell you'll go through. There's a program, for authoring eBooks, called Sigil, that's both perfect and without alternatives. Unfortunately, the program's author chose to use dependencies, instead of his own code, for myriads of features. If your distro's version of Sigil is buggy, or your distro doesn't even offer it, you've got a white-knuckle, trial and error couple hours of experimental compilation in front of you to make Sigil work, and most likely you'll need to explore lots of source codes to figure out the compile errors.

It didn't have to be this way. I'm pretty sure this app could have been written with dependencies on only an XML parser/writer library, a GUI library, and an internationalization library. I'm pretty sure Qt4 could have done the GUI and internationalization. Annoyingly, modern (less than a couple years old) versions of Sigil require Qt5. How many distros, or at least their stable versions, make Qt5 available? Certainly not Debian Wheezy. I had to compile an older version on Wheezy, and felt lucky to be able to do that.

Sigil's original author got tired of maintaining it and moved on to other things. Hey, it happens all the time, no problem. But with its gratuitous dependencies, very few people were willing and able to continue maintaining it. Luckily, somebody took over the project, but I'd sure feel better about it if it were something a mere mortal could maintain if there were no other alternative. But the gratuitous dependencies pretty much rule out mere mortals.

All the time, you hear people pontificate "Don't reinvent the wheel" when you code with minimal dependencies, or reject software with promiscuous dependencies. Never reinventing the wheel means making cobbled-together bugware.

Not Invented Here

Let's get back to init systems, specifically, systemd. Not only are some distros depending more and more on systemd, but now some of them even do things "the systemd way", and if you don't want systemd, you need to go in and redo things.

For instance, systemd has managed to subsume much of your computer's audio, and has removed the necessity for /etc/asound.conf and $HOME/.asoundrc. So when you switch to Epoch, you get bizarre and unhelpful error messages about missing devices, missing files and the like. For the record, my /etc/asound.conf follows:

defaults.pcm.card 1
defaults.ctl.card 1

And my ~/.asoundrc looks like the following:

pcm.!default {
        type hw
        card 1
}

ctl.!default {
        type hw
        card 1
}

I'm good at troubleshooting, but not knowing exactly the way the Linux audio system works (few do), I was reduced to web searching error messages. So I'm left scratching my head wondering why systemd feels it must take over every single thing it touches. Is it to make jumpering around systemd more difficult, and if so, what is the motive.

Undiscoverable Software and the Cost of Bad Documentation

Can somebody please tell me why the Wicd man page doesn't list the syntax for running it as a server? Seriously, the Wicd project's been around for a long time: Could not somebody in the project, even if it had to be the developer, show the syntax, and explain exactly how it must be wedded to dbus?

Not to pick on dbus-dependent software specifically, can somebody explain the ambiguous and incomplete man pages and documentation for wpa_supplicant, dhcpd, and dhclient? I'm curious why the wpa_supplicant.conf man page does its job by listing a few examples, and fails to include an example for a no-security Wifi hotspot like they have in half the restaurants in the country?

Look at the following snippet, straight out of the wpa_cli man page, describing the set command in the command list:

set    set variables (shows list of variables when run without arguments)

In the preceding, see the text saying that if you use the command set without arguments, it shows the variables that you can set? Well, on my Manjaro 8.11 box, that's just plain wrong:

[root@litttmanjaro etc]# wpa_cli set
Selected interface 'wlo1'
Invalid SET command - at least 2 arguments are required.
[root@litttmanjaro etc]#

It's a shame, because the wpa_cli program would have been perfectly suited to discoverability. For instance, running wpa_cli could have produced something like this:

# wpa_cli
Info on settings changes: wpa_cli set -h
Connection Information: wpa_cli connect -h
Connection Commands: wpa_cli commands -h
Miscellaneous: wpa_cli misc -h

Perhaps some of the help screens brought on by those help commands could have sub-helpscreens. The point is, the user could drill down and discover how to do what needs to be done. Instead, it has a man page that's incomplete and in at least one place wrong, it has one -h help screen 152 lines long, so that if the user really wants to use it, the user must do detailed web research, and we all know there's as much wrong info on the web as right information. An opportunity missed.

The wpa_supplicant constellation of software is outstanding. In fact, Wicd uses that software, binding it to dbus, which binds it to systemd unless you take a lot of steps. The trouble with the wpa_supplicant constellation is that its ambiguous documentation and lack of discoverability mean that in the hands of the non-expert, it's bad software. So the user is tempted to use the NetworkManagers of the world, and gets locked into package manager enforced dependencies.

It doesn't have to be this way, and I'm doing my best to change the situation.

Initramfs / Initrd

Initramfs and initrd files are used to give you a filesystem from the very start of the boot. In the old days, they weren't always necessary, but with this hip new hobby of having directories /sbin, /bin, and /usr/sbin, be symlinks for /usr/bin, initramfs files are more necessary than ever. So like it or not, you need to understand them, and one of the best ways to understand them is to take them apart and put them back together.

There's a difference between the older initrd files and the newer initramfs files. The older initrd files were block device images that could be loopback mounted to produce a directory tree. The newer initramfs files are gz compressed cpio archives of a directory tree, which the kernel knows how to use. Grub has an initrd line identifying the initramfs file or the initrd file, and as I said, most modern distros go for the initramfs option.

Taking Apart an Initramfs

NOTE:

This subsection uses the initramfs-312-x86_64.img from a standard Manjaro 8.11 64bit Systemd Edition installation. Your tree might look different, but the principles should be the same.

This subsection assumes an initramfs file named initramfs-312-x86_64.img is in the current directory. After logging in as root and copying initramfs-312-x86_64.img to /root, perform the following steps to take it apart:

root@mydesq2:~# mkdir myfs
root@mydesq2:~# cd myfs
root@mydesq2:~/myfs# cp ../initramfs-312-x86_64.img z.gz
root@mydesq2:~/myfs# gunzip z.gz
root@mydesq2:~/myfs# cpio -i --make-directories < z
22042 blocks
root@mydesq2:~/myfs# rm -f z
root@mydesq2:~/myfs# ls -ldF *
lrwxrwxrwx 1 root root     7 Jan 11 10:19 bin -> usr/bin/
-rw-r--r-- 1 root root  2513 Jan 11 10:19 buildconfig
-rw-r--r-- 1 root root    87 Jan 11 10:19 config
drwxr-xr-x 3 root root  4096 Jan 11 10:19 dev/
drwxr-xr-x 5 root root  4096 Jan 11 10:19 etc/
drwxr-xr-x 2 root root  4096 Jan 11 10:19 hooks/
-rwxr-xr-x 1 root root  2300 Jan 11 10:19 init*
-rw-r--r-- 1 root root 12469 Jan 11 10:19 init_functions
-rw-r--r-- 1 root root  2567 Jan 11 10:19 keymap.bin
-rw-r--r-- 1 root root     0 Jan 11 10:19 keymap.utf8
lrwxrwxrwx 1 root root     7 Jan 11 10:19 lib -> usr/lib/
lrwxrwxrwx 1 root root     7 Jan 11 10:19 lib64 -> usr/lib/
drwxr-xr-x 2 root root  4096 Jan 11 10:19 new_root/
drwxr-xr-x 2 root root  4096 Jan 11 10:19 proc/
drwxr-xr-x 2 root root  4096 Jan 11 10:19 run/
lrwxrwxrwx 1 root root     7 Jan 11 10:19 sbin -> usr/bin/
drwxr-xr-x 2 root root  4096 Jan 11 10:19 sys/
drwxr-xr-x 2 root root  4096 Jan 11 10:19 tmp/
drwxr-xr-x 6 root root  4096 Jan 11 10:19 usr/
-rw-r--r-- 1 root root     2 Jan 11 10:19 VERSION
root@mydesq2:~/myfs#

You can now explore this tree. Look at the init file. Browse /usr/bin to see which executables your computer actually needs at the beginning of its boot.

View the scripts in the hooks/ directory, especially hooks/udev. Then view init_functions, and see that its run_hookfunctions() function simply loops through hooks/, running each script. Observe that the init script imports init_functions and calls function run_hookfunctions(). The bare-bonesness of this tree is actually a handy way to learn a little more about Linux.

Here's another little mystery solved. The initramfs' init script calls the fsck_root() function, which, if you look in init_functions, calls fsck_device if not $fastboot. After init runs fsck_root, it then mounts the root device. This is how your computer does an fsck on the root device before mounting it as /new_root. So the fsck happens on an unmounted device, which of course is mandatory if you don't want to trash that device. All of this happens before your actual disk's /sbin/init gets run.

Speaking of which, the last command in the initramfs' /sbin/init runs the switchroot program, which does the following:

  1. Recursively deletes everything under /.
  2. Makes /new_root becomes the new /.
  3. Runs the new root's /sbin/init.

Pretty slick, huh?

It's geeky cool, and therefore fun. But of course the problem is, if you happen to lose your initramfs file, you're dead meat, because it's way too complex to rebuild from scratch. Also, having an initramfs just makes it tougher to use Grub-Foo to muscle your way back into a system that's lost its MBR.

But of course, to initramfs or not to initramfs isn't your call, it's your distro's call, and more and more are making initramfs mandatory. The reason I even wrote this section is because, if you know how to take apart and study your initramfs, you have a better Mental Model of your early boot process, so you're more able to diagnose early boot problems.

Putting an Initramfs Back Together

I don't advocate changing your initramfs. Unless you just want to get geeky with it, the risks greatly outweigh the potential benefits. And for gosh sakes, if you change your initramfs, keep the original, so you can still boot using the original just by changing the initrd value in your Grub command line.

Anyway, the reassembly process is pretty much the opposite of taking it apart. As root, start in the myfs directory you made while taking your initramfs apart.

root@mydesq2:~/myfs# find . | cpio -H newc -o > ../neww 22041 blocks root@mydesq2:~/myfs# cd .. root@mydesq2:~# gzip neww root@mydesq2:~# mv neww.gz neww.img root@mydesq2:~#

As an experiment, I performed the putting back together without changing anything, and the new initramfs was a different size than the original. But, when I then took the new initramfs apart into a different directory (myfs2), and performed a diff -r, the two directories proved to be identical.

Tiny Experimental Initramfs

It's also possible to boot to a tiny initramfs without ever involving the hard disk (other than reading the initramfs off the hard disk). If you're interested in that kind of thing, here is some more info on the subject.

Initramfs and Troubleshooting

Sometimes you need to know very early boot information in order to troubleshoot init problems. When you have such needs, there's no substitution for being able to see what the initramfs is really doing. This section has shown you how to take apart your initramfs in order to find this early boot information.

The Second Battle of Red Hat

The last time I took the fight to Red Hat, I was defeated by LVM problem and didn't get very far. This time, I installed a fully Ext4 system, and once again initted it wih Epoch. The results were much more pleasing...

Instead of installing the default CentOS, I used Ext4 throughout, and didn't install LVM. The disk mounts follow:

[slitt@localhost ~]$ 
mount | grep ^/dev

/dev/sda2 on / type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)
/dev/sda3 on /home type ext4 (rw,relatime,data=ordered)
[slitt@localhost ~]$

As far as I can tell, the preceding reboots perfectly, with no journal rejigger on reboot.

The following is the /etc/epoch/epoch.conf I used to init the system:

BootBannerText=Sorry, Lennart!
BootBannerColor=CYAN

Hostname=FILE /etc/hostname
DefaultRunlevel=boot
EnableLogging=true
DisableCAD=true
BlankLogOnBoot=true
MountVirtual=procfs sysfs devpts+ devshm+

# DEFINE PRIORITY CONSTANTS
DefinePriority=Rwfs_Start 30
DefinePriority=Mountall_Start 35
DefinePriority=Early_Getty_Start 40
DefinePriority=Sysclock_Start 35
DefinePriority=Udev_Start 20
DefinePriority=Network_Start 50
DefinePriority=Dev_Mixer_Start 60
DefinePriority=Typical_Daemon_Start 80
DefinePriority=Typical_Getty_Start 70
DefinePriority=Display_Manager_Start 120


DefinePriority=Display_Manager_Stop 40
DefinePriority=Typical_Daemon_Stop 10
DefinePriority=Sysclock_Stop 80
DefinePriority=Killall5_Soft_Stop 90
DefinePriority=Killall5_Stop 91
DefinePriority=Rwfs_Stop 100



ObjectID=getty2
        ObjectDescription=Early getty on /dev/tty2
        ObjectStartCommand=agetty tty2 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Early_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

# MAKE readwrite just after udev instantiates,
# MAKE readonly again after all killalls
ObjectID=rwfs
        ObjectDescription=root filesystem read-write support
        ObjectStartCommand=/usr/bin/mount -o remount,rw /
        ObjectStopCommand=/usr/bin/mount -o remount,ro /
        ObjectStartPriority=Rwfs_Start
        ObjectStopPriority=Rwfs_Stop
        ObjectEnabled=true
        ObjectRunlevels=boot

# MOUNT ALL just after rwfs
ObjectID=mountall
        ObjectDescription=Mount all filesystems
        ObjectStartCommand=/usr/bin/mount -a
        ObjectStopCommand=NONE
        ObjectStartPriority=Mountall_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectRunlevels=boot

# KINDLY request all processes to end themselves
# Give them 2 seconds to do so.
ObjectID=killall5_soft
        ObjectDescription=Terminating all processes
        ObjectStopCommand=killall5 -15 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=Killall5_Soft_Stop
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

# MURDER all remaining processes
# Give them 2 seconds to do so.
# Then hand control to service rwfs to
# remount / read-only to prevent filesystem corruption
ObjectID=killall5
        ObjectDescription=Killing all processes
        ObjectStopCommand=killall5 -9 && sleep 2
        ObjectStartPriority=0
        ObjectStopPriority=Killall5_Stop
        ObjectEnabled=true
        ObjectOptions=HALTONLY RAWDESCRIPTION

ObjectID=sysclock
        ObjectDescription=Configuring system clock
        ObjectStartCommand=hwclock -s
        ObjectStopCommand=hwclock -w
        ObjectStartPriority=Sysclock_Start
        ObjectStopPriority=Sysclock_Stop
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN simple shellscript to get udev running
# Udev is necessary to start network
# Run it before remounting / read-write.
# Don't bother to stop it at all
ObjectID=udevd
        ObjectDescription=Starting udev
        ObjectStartCommand=/root/udev_start.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=Udev_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# RUN shellscript to set up network. Works on any distro having
#  the ip command. No config files necessary.
#  start it after Udev is up and / is read-write
#  and the early Getty is running
ObjectID=network
        ObjectDescription=Setting up net devices and the network
        ObjectStartCommand=/root/upnet.sh
        ObjectStopCommand=NONE
        ObjectStartPriority=Network_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

# LOAD kernel module for /dev/mixer, without ALSA or PULSE
# Run after early Getty and network, but before daemons
# No need to stop it on shutdown.
ObjectID=devmixer
        ObjectDescription=Setting up /dev/mixer via oss
        ObjectStartCommand=/usr/bin/modprobe snd_mixer_oss
        ObjectStopCommand=NONE
        ObjectStartPriority=Dev_Mixer_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=RAWDESCRIPTION
        ObjectRunlevels=boot

ObjectID=getty3
        ObjectDescription=Getty on /dev/tty3
        ObjectStartCommand=agetty tty3 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty4
        ObjectDescription=Getty on /dev/tty4
        ObjectStartCommand=agetty tty4 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty5
        ObjectDescription=Getty on /dev/tty5
        ObjectStartCommand=agetty tty5 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

ObjectID=getty6
        ObjectDescription=Getty on /dev/tty6
        ObjectStartCommand=agetty tty6 &
        ObjectStopCommand=NONE
        ObjectStartPriority=Typical_Getty_Start
        ObjectStopPriority=0
        ObjectEnabled=true
        ObjectOptions=SERVICE AUTORESTART
        ObjectRunlevels=boot

# MANAGE sshd daemon
ObjectID=sshd
        ObjectDescription=Manage sshd daemon
        ObjectStartCommand=/usr/sbin/sshd
        ObjectStopCommand=PIDFILE /run/sshd.pid
        ObjectStartPriority=Typical_Daemon_Start
        ObjectStopPriority=Typical_Daemon_Stop
        ObjectEnabled=true
        ObjectRunlevels=boot
        ObjectOptions=SERVICE AUTORESTART

# MANAGE crond daemon
ObjectID=crond
        ObjectDescription=Manage crond daemon
        #ObjectStartCommand=/usr/bin/crond
        ObjectStartCommand=/usr/sbin/crond
        ObjectStopCommand=PIDFILE /run/crond.pid
        ObjectStartPriority=Typical_Daemon_Start
        ObjectStopPriority=Typical_Daemon_Stop
        ObjectOptions=SERVICE AUTORESTART
        ObjectEnabled=true
        ObjectRunlevels=boot

ObjectID=gpm
	ObjectDescription=GPM Mouse Server
	ObjectStartCommand=gpm -m /dev/input/mice -t imps2
	ObjectStopCommand=PIDFILE /run/gpm.pid
	ObjectStartPriority=Typical_Daemon_Start
	ObjectStopPriority=Typical_Daemon_Stop
	ObjectEnabled=true
	ObjectOptions=SERVICE AUTORESTART
	ObjectRunlevels=boot

The preceding calls several shellscripts, starting with /root/run_devfs.sh:

#!/bin/sh
/usr/lib/systemd/systemd-udevd --daemon
/usr/bin/udevadm trigger --action=add --type=subsystems
/usr/bin/udevadm trigger --action=add --type=devices
/usr/bin/udevadm settle

The preceding shellscript starts up and augments udevd, which is necessary for network devices. Next is the /root/upnet.sh shellscript, which is how you configure a fixed IP address Ethernet interface:

hostname -F /etc/hostname
ip link set dev lo up
ip link set dev enp3s6 down
ip addr add 192.168.100.87/24 dev enp3s6
ip link set dev enp3s6 up
ip route add default via 192.168.100.96

The bottom line is this: I have an Epoch-booting CentOS that works pretty darn well. The shutdown probably needs a little work so that the journal doesn't get rewritten on startup, and it doesn't do "boot directly to GUI" right, mostly because I don't boot directly to GUI anyway. But I proved my point. I took an alternate init to Red Hat's most sacred territory, and won.

What this means for you is that neither Red Hat, Lennart Poettering, or the Debian CTTE have any say over your init system. There are several init systems you can put on any computer, including those most dear to Red Hat.

1/30/2015: So Long For Now

It's been three weeks since I wrote anything in Manajaro Experiments, because my itch is scratched. Going into the Manjaro Experiments, I was very afraid that, in order to have a productive desktop, I'd need to leave free software entirely, and use a Mac. The Manjaro Experiments have convinced me that I'll have no problem replacing my Debian Wheezy daily driver desktop with the distro of my choice, sporting the init of my choice.

I do want to do two more init experiments:

  1. Reproduce Karl Hammar's Busybox init
  2. Init via Suckless Tools' Suckless Init.

But I'll wait a little bit. Here's the thing: The Manjaro Experiments were a response to an existential threat to my Linux usage. But systemd was not the only existential threat. The once proud LyX project has seen fit to ignore semantically correct ePub export, and ePub maker Sigil doesn't make PDFs. Also, Sigil is constantly under threat of abandonment, and its dependencies are almost as obnoxious as those of systemd. So, to be guaranteed a continuing ability to make books destined for Print, PDF andePub, I'm making my own book authoring system, called Stylz, and that's where all my spare time goes these days.

So long, for now.