Troubleshooters.Com®,
Linux Library
and Init System Choices Present:
Init System Features and Benefits
Copyright © 2015 by Steve Litt
CONTENTS:
Just so we're all on the same, a feature is a trait or property of the system. A potential benefit is a change in your life. A benefit is an improvement in your life. A potential benefit becomes a benefit if and only if the change makes your life materially better. For instance, fast booting is a potential benefit, but it becomes a benefit if and only if either:
In other words, if you boot your personal desktop once a week, you don't really care whether it boots in four seconds or four minutes. If you boot it every morning, you don't care whether it takes 4 seconds or 30 seconds. Now let's add in features.
Features are traits or properties of the system, presumably for the purpose of bestowing benefits on the user or owner. A specific potential benefit can be realized by alternate features. For instance, parallel daemon instantiation can make for faster booting. And so can not running scripts and daemons unimportant to you. And, if you're booting straight to GUI, so can a lightweight window manager.
Here's an example: Both a magnesium paddle shifter in a car, and parallel process starting in an init system, are features. The potential benefit of the magnesium paddle shifter is faster shifting of an automatic transmission, while the potential benefit of parallel process starting is faster boots. The paddle shifter becomes a true benefit if your engine has too little oomph to correctly accelerate you when your transmission shifts for itself, or if you're trying to impress somebody. Faster boots become a true benefit when you need to perform a lot of boots, or when you need very high availability, when you need to boot for a presentation right now, or when you want to impress someone.
The bottom line is this: A potential benefit is relevant if and only if it substantially betters your life. A feature is relevant if and only if it produces potential benefits leading to true benefits. Always remember this when you hear others extol the benefits of their chosen init system.
Examine the following feature matrix, with features going down the left side, and the most common inits going across the top:
| Epoch | nosh | OpenRC | perp | runit | s6 | systemd | sysvinit | Upstart | uselessd | ||
| True PID1 init | Y | Y | N | ? | Y | Y | Y | Y | Y | Y | |
| Respawning | Y | Y | N | ? | Y | Y | Y | Y | Y | Y | |
| Parallel daemon startup | N | Y | Y | ? | Y | Y | Y | N | Y | Y | |
| Process Dependency model | numeric | script | calc | script | script | script | calc | numeric | ? | calc | |
| Event based? | N | ? 2 | Y | ? | ? | ? | Y | N | Y | Y | |
| OS Toolkit | N | N | N | N | N | N | Y | sort of | N | N | |
| Socket activation | N | Y 2 | N | N | N | N | Y | N | Y ? | Y ? | |
| Daemontools Inspired | N | Y | N | Y | Y | Y | N | N | N | N | |
| (Subjective) grade for online documentation 4 |
A- | C- | B 1, 5 | D | C+ | C- 3 | C 5 | C 5 | B- 5 | F | |
| Declarative Syntax | Y | N | N | N | N | N | Y | ? 6 | Y | Y | |
| Can natively run one-time processes |
Y | N | Y | N | N | N | Y | Y ? | Y ? | Y ? | |
| License | unlicense .org |
? 8 | BSD 1011 | ? 9 | BSD 7 | ISC | LGPL 11 | GPL | GPL 11 | LGPL ? | |
| Primary install method | Compile | Comp ile |
Package | Comp ile |
Comp ile |
Comp ile |
Package | Package | Package | Compile | |
| Cgroups | N | ? | Y | ? | ? | ? | Y | ? | ? | Y | |
| supervision-scripts compatible |
N | N | N | N | Y | Y | N | N | N | N | |
I only counted online documentation: Few people will download a project in hopes that docs from the tarball will be better or more available than that available on the 'Net. Findability counts: Docs linked straight from the projects home page get the nod over longer navigations, and get even more of a nod over third party documentation (like what you're reading right now). Inits likely to be installed by a package manager were given a pass from having to explain installation and configuration, so it's somewhat of an apples and oranges situation. Docs requiring trips through github were rated lower: Who wants to do that.
My perception is that the greatest single failure of most of these init systems was insufficient documentation for a mere mortal to easily get them up and running.
Completely apart from "event driven", there are three common process dependency models:
Numeric process dependency models rely on the admin or packager to guess the order in which processes should be run. Numeric process dependency is appropriate only when process startup is sequential (not parallel) and there are not a great many processes being run. In other words, you wouldn't use Epoch to init a machine spawning fifty daemons. Why you'd want fifty daemons is quite another question.
Calculated dependencies are when the init system calculates the run order on the basis of each process' "requires", "after", and "provides". If events are not a consideration, a fairly simple Python program could convert calculated dependencies to numeric dependencies, but why bother? The power of calculated dependencies is that with the right init system, they can be merged with events to greatly reduce race conditions. Calculated dependencies are excellent for event driven inits with many spawned processes.
Script based dependencies are good in the same use cases as calculated dependencies. They're especially useful in daemontools-inspired inits such as runit, s6, nosh and perp, all of which retry spawning the process until it succeeds. Script based dependencies are based on this approximate algorithm:
if dependent process not running: again spawn the dependent process return failure on the current process spawn the current process
Note that if the current process has several dependencies, there will be several of those tests, spawns and failure returns. Note also that in certain use cases you can't use the preceding logic, because it will over-and-over again spawn the dependent process. The runit init system has a special per-process file, called ./check, whose purpose is to detect a fully functional dependency and do the right thing otherwise.
Obviously, the dependency model I just described depends on polling, whereas event-driven dependency models don't poll: An advantage for event-dependency. That being said, depending on use case, it's not that much of an advantage, and of course polling is usually simpler and easier to debug than events.
Obviously, the dependency model I just described depends on polling, whereas event-driven dependency models don't poll: An advantage for event-dependency. That being said, depending on use case, it's not that much of an advantage, and of course polling is usually simpler and easier to debug than events.
Daemontools-inspired inits are all built to manage (respawn and control) processes, meaning that when such a process ends, it gets started again. Most of the daemontools-inspired inits I'm aware of can run single-shot scripts in their stage 1 (startup) and stage 3 (shutdown) stages, but not their stage2 (management) stage, which runs whenever the computer isn't either booting up or shutting down.
This presents a problem, because sometimes you want a crashed process to stay crashed (with appropriate notification), and sometimes you want a one-time process, that would normally be done during boot, to happen after some stage 2 processes are up and running.
I can think of several ways to make stage 2 processes run once. They're not aesthetically beautiful, they'd hand over great talking points to anti-daemontools people, but they'll work. Better news still, right now, as I write this, smarter people than I are working on ways to solve this problem elegantly.
Both Epoch and sysvinit can intermix respawning and one-time processes, and OpenRC runs only one-time processes. My assumption is that systemd, uselessd, and upstart can intermingle respawning and one time processes, but I have no documentary proof.
Runit, s6, perp and nosh were all inspired by daemontools, a respawning process management tool with a surprisingly simple and understandable architecture based primarily on the Unix filesystem. Daemontools, along with the inits it inspired, employs very simple run scripts to daemonize a foreground process. Daemontools and the inits have kludges to try to daemonize software that cannot be run in the foreground, but the right way to use these inits is to run the program in the foreground. So for instance, sshd -d or cron -f
Inits like sysvinit and OpenRC give init scripts a bad name, but in fact daemontools-inspired programs usually have incredibly simple run scripts that should not in any way be compared to those of sysvinit and OpenRC. They're simple enough that a half way intelligent admin could write them from scratch. But writing them from scratch isn't usually necessary, because of facilities like supervision-scripts, a set of process run scripts, for many common programs, portable between runit, s6, and daemontools itself. Avery Payne told me on 1/2/2015, to include the following warning when describing supervision-scripts:
WARNING:
Please note the scripts are still pre-0.1, and have not been 100% tested. Many still have an "untested" marker in their service definitions; use at your own risk. I can only say that I run a subset of these on my home server under runit and they work for me. I am still working on testing them under all three frameworks.
The preceding warning doesn't subdue my enthusiasm, because once supervision-scripts is tested and working, people will be able to use runit or s6 without worrying about "translating sysvinit scripts." Not only this, but I'd guess this will become very popular with "upstreams" because they'll no longer be responsible for their software's init scripts, at least not if the init system is runit or s6. This is how things always should have been.
Daemontools-inspired inits have script based process dependency handling. This is superior to numeric dependency handling, more difficult to set up than calculation based dependency handling, but also more versatile than calculation style dependency handling.
The daemontools-inspired inits are simple, admin-friendly, very efficient, fast booters, easy to install without a package manager, versatile, DIY friendly, and rock solid.
Your mileage may vary, but in my opinion respawning should be an option. With Epoch, it is. You can declare anything to be either respawned or not. Same with sysvinit. I'd assume it's the same with systemd, upstart and uselessd.
OpenRC cannot respawn at all, and the daemontools-inspired inits are designed to respawn, so special steps must be taken to get them not to respawn.
The main rationale for socket activation and event based init is the modern Linux kernel's parallel and indeterminate instantiation of various things. Apparently the kernel issues events after each instantiation is complete, at which time an init process depending on that instantiation can launch.
The potential benefit is that race conditions don't cause failures. However, there are many other features, both within the init system and without, that can almost completely eliminate such race condition caused failures. An obvious one is a short sleep. I haven't tested this on huge numbers of computers and use cases, but I have a strong feeling that a simple ten second sleep at the beginning of init would allow the kernel to complete all its instantiations, except those that depend on something the init spawns. So, if you can tolerate adding ten seconds to the boot (the time to sip your coffee and chew one bite of a Danish roll), you don't need socket activiation or event based init. I have a hunch that in most situations it will add up being more like two seconds, but even ten seconds is an amount whose only detriment is degradation of bragging rights.
NOTE:
Keep in mind that /sbin/init or /usr/bin/init don't run until after the kernel runs the init in the initramfs, and that initramfs init calls the hard disk's init, which usually happens at the very end of the initramfs' init. If there's a time consuming fsck, that happens during the initramfs' init before the call to the hard disk's init. So the bottom line is this: there's plenty of sequential activity that happens before what you think of your init even starts, and this should provide a buffer against kernel/init race conditions.
Another substitute for socket activation and event based init is daemontools style run scripts that test for a kernel process being fully functional. If the kernel process is designed not to run until it gets a message from something spawned by init, such a daemontools like run script could even signal the kernel.
The other thing is, if you're running an edge case causing kernel process to take several seconds to get running, perhaps that edge case itself should be investigated. To get more information, I just rebooted my Epoch-initted Centos box, a four year old box with an Asus M4A785-M mobo, an AMD Athlon(tm) II X2 250 Processor, and 4GB RAM. In other words, it's no racehorse. Nevertheless, it took only four seconds to get from Grub to the start of Epoch, and another four seconds for Epoch to boot it to a complete CLI system. In other words, the kernel and the initramfs init program took four seconds to do their job. If the kernel is still trying to start processes at, let's say, 10 seconds after it takes control, I'd sure consider the possibility that something that needs investigation.
Socket activation/event based, plus parallel process instantiation, is how you do it when your top three priorities are boot speed, boot speed and boot speed. Otherwise, there are many ways to avoid race conditions with modern, semi-indeterminate Linux kernels.
There are two ways you can obtain your init system: The package manager, and compile-it-yourself. In 2015, up to date distros bestowed by your package manager will pretty much be limited to systemd, OpenRC and sysvinit. All the rest, at least for most distros, you'll need to compile, install and configure yourself. When you need to do it yourself, good documentation can speed the install/configure process by an order of magnitude or more. This is why, in the Manjaro Experiments, I succeeded in installing Epoch and runit, but failed with s6 and nosh. Epoch and runit have docs that better cover their installation and configuration procedures.
Speaking of documentation, Manjaro Experiments itself is excellent documentation on inits in general, and the process of becoming expert with inits in general. Read it. I also highly suggest you start with an experimental Manjaro/OpenRC setup, and experiment.
Respawning is the ability to automatically rerun a process that fails. So if your web server goes down, it gets started up again whether your admin is in the server room or on a beach in Tahiti. This isn't particularly important, because often that's not the behavior you want on a broken service, and also because if your init can't respawn, you can run something like daemontools that can.
A more important benefit from respawning is the ability to keep trying until you succeed, as in the instantiation method of the daemontools-inspired inits. These inits can be scripted to check for dependencies, run the dependencies and exit fail if the dependencies aren't up, and try again in a few seconds. This is a very effective and efficient polling method that substitutes for event based instantiation. However, even this benefit can be simulated (kludgily) with shellscripts in other inits, and of course in event based inits it may not even be necessary.
Personally, I don't think an init's inability to respawn, or its inability to natively and simply run once, is a showstopper.
Parallel Process Startup has exactly one potential benefit: Faster boots. If boot speed were not a priority for you, you could boot sequential startup Epoch or sysvinit, with appropriate sleeps numeric order. The cost of parallel startup is increased complexity. Your job is to decide whether it's worth it.
This cost/benefit analysis requires you to first know the boot times of sequential startup inits. For instance, Epoch, which starts services sequentially, boots to full CLI readiness, on my experimental CentOS system, in 8 seconds. Systemd boots the same system to X (without window manager) in 4 seconds. Personally, I wouldn't start worrying about boot speed until boot time exceeded 30 seconds, so in my use case, faster boot isn't a benefit, and therefore sequential startup isn't a feature I need. Obviously, it would be very different if I had a contract specifying no more than six minutes per year downtime.
There's yet another way to gain the benefit of a fast boot. You can take a machete to all the edge case CYA processes that get run by your distro's standard init. The standard list of processes are meant to cover use cases you don't have: You can safely remove a lot of them.
Another way to gain this benefit is to profile your initialization. Which processes take several seconds to start, and why? Are they performing complex handshaking, like dhclient? If so, what can you do to make that faster? Are they timing out waiting for reverse DNS before finally starting? If so, make sure that your reverse DNS works, and it works before such a process gets run. Is it taking 20 seconds to start Gnome once you log into your Display Manager? There's an app for that: It's called Openbox or LXDE.
Depending on your viewpoint, this is either overhyped marketing or the most DIY-friendly feature in the world.
Bias Disclaimer
I have always had very, very good feelings toward the daemontools way of administering processes. I'm not objective about daemontools-inspired inits.
The cool thing about the daemontools-inspired inits is that, in many cases, you do nothing but a simple ./run script for each process. This is welcome news for refugees from the gargantuan script files of sysvinit and OpenRC. It might even be a welcome change from the unit and install section options in systemd and uselessd, although I'd imagine most of the work could be done by requires and after.
Another outstanding benefit of daemontools-inspired inits is that, with two minutes work, you can turn every output of the process being run, both stdout and stderr, into a timestamped log file.
But I think the reason I and other people love daemontools-inspired inits is that, from a Unix viewpoint, they just make sense. Their "database" is nothing more than a couple directory trees and a few short shellscripts (or scripts in special shell-like languages). You see them, you immediately understand what they do, and they just make sense. Oh, and they perform well too.
And don't forget, if Avery Payne's supervision-scripts project succeeds, at least the runit and s6 members of the daemontools brigade will also become almost trivial to administer.
Lennart Poettering enthusiastically declares systemd an "OS Toolkit" rather than an init. My reading of and listening to Poettering indicates this means their project is making all sorts of OS building blocks to replace formerly independent things like udev and consolekit, the idea being that you can build an operating system by bolting a few of those things together.
I believe that, when the echos of rhetoric finally fade, it will have been this one feature that caused, and will continue to cause, extreme hostility toward systemd. Not the hundreds of thousands of lines of code. Not binary log files that can be easily rewritten as text. Not even the "wontfix" bugs. Those things were talking points. The real cause of hostility, in my opinion, was the OS Toolkit issue.
A great many Linux users liked the DIY opportunities bestowed by independent, relatively narrow interface parts like udev, consolekit, and several others. Systemd even goes so far as to offer hooks for desktop environments. This is why few hold any animosity toward uselessd, a knockoff of the init part of systemd, but without the "OS Toolkit" ambitions.
So, if you want to build an entirely new Linux, with minimal effort, by bolting together a few systemd offerings, then systemd's "OS Toolkit" feature offers a spectacular benefit. Oppositely, if you want to be able to change the functionality of your computer by inserting parts, replacing parts, even removing parts, then systemd's "OS Toolkit" feature bestows a harsh disbenefit. If you just use your computer and don't care about building from scratch or DIY modifications, and just want to use your computer the way your distribution gave it to you, then the OS Toolkit feature probably doesn't matter one way or the other.
Declarative syntax means configuring your processes primarily with key-value pairs rather than with scripts. If you've been slogging around with the megascripts used by sysvinit and OpenRC, this probably sounds like an excellent idea right about now. If you've been using a daemontools-inspired init, you're probably saying "you'll pry my short and versatile scripts out of my cold, dead fingers."
I've used declarative Epoch, and I've used scripted runit, and to tell you the truth, if either declarative syntax or scripts are interfaced reasonably by the init, they're both fine. You can get to the same benefits, both ease and versatility, either way.
Of course, you can install any software by compiling, and you can install any software with a package, although in some cases you might need to make that package yourself. Oddly, or perhaps not so oddly, in the init world of early 2015, it turns out to be an either/or proposition. For any init, it's either overwhelmingly compiled, or overwhelmingly installed via the package manager. Each of these two opposite features has its own benefit.
The benefit for installing via package is pretty obvious. It's (theoretically) a five minute deal with apt-get install or packman -S or yum install. And the installed init works in harmony with the rest of your software. No muss, no fuss, no bother.
And there's a disbenefit: What if your distro's packagers stop offering your favorite init, or offer you a broken one? You need to start a search for a new distro. Or you need to compile your own init. This is not academic. Debian's original idea to offer only systemd instead of sysvinit caused the Debian init wars, and the init wars caused Debian to (perhaps temporarily, we'll see) continue to offer a sysvinit package.
The do it yourself compile install method offers two benefits:
A word about the simultaneous multiple inits. This is more useful than you might first imagine. Just like many people have a fallback kernel for when things go bad, multiple inits give you one or more fallback inits when things go bad. With multiple inits, if you ever wonder if a problem is being directly or indirectly caused by your init, you can just change your kernel line in Grub, reboot, and compare your system using the two different inits. If you accidentally bork an init (I've done that several times), instead of whipping out the time consuming System Rescue CD, you just boot to the other init, you fix your problem, and boot back into your normal init.
By the way, the reason I list simultaneous functional inits as a benefit of Compile It Yourself is because, generally speaking, init packages disable each other.
A word about why OpenRC, systemd, sysvinit and Upstart have packages and the rest don't. The four in the preceding sentence are probably four of the five most complicated init programs there are, the fifth being uselessd. The likelihood of even tech-savvy users being able to compile those four by themselves is fairly low. Meanwhile, for some reason, those four are or have been them most mainstream.
On the other hand, Epoch, runit, s6, perp and nosh tend to be simpler and probably easier to compile (though I was unable to compile nosh). As a matter of fact, Epoch's #1 design priority was minimal dependencies, making it an easy compile candidate on any computer with a Linux kernel. And for some reason, these five inits have traditionally been ignored by distros. So you have no choice but to compile them, and they're easy to compile.
One more thing: There's no law saying you can't have a package-bestowed init coexist with one or more Compile It Yourself inits. Personally, I think that's the best of both worlds.
Cgroups, or "Control Groups", is a Linux kernel feature that is taken advantage of by (at least) OpenRC, systemd and uselessd. It's a better way of managing running processes, and is used extensively in container software like Docker.
From what I hear, using cgroups is better for killing zombies and eliminating the need for doubleforking. The potential benefit is more control. Personally, I don't think its worth the cost of a complex init system: Your mileage may vary. Also, I'd imagine it's pretty easy to use cgroups with non-cgroup-aware inits using cgroup binaries. Debian Wheezy has something called cgroup-bin for all these binaries. CentOS and Manjaro don't, but I'm thinking it might be possible to code them yourself, or grab the source from somewhere and compile it for your own distro.
Two of the listed inits are compatible with supervision-scripts, which is a pre-made bunch of run scripts compatible with runit and s6. Potential benefits include:
Before you take to the streets cheering, keep in mind that supervision-scripts is in its infancy, still under very heavy design and development, and it's a moving target. Today, supervision-scripts is very much a "use at your own risk" type thing.
But if and when supervision-scripts fulfills its mission of providing startup scripts for all common daemons, then not only will runit and s6 be very high quality, but they'll also be dead-bang easy, whether via a package manager or self-installed.
The past several months provided heated discussions of init systems. Many init debaters have based their arguments on the presence or absence of a particular feature. Such arguments failed to persuade, because in their hearts, everyone knows the only purpose of features is to provide benefits, and different use cases require different benefits, and a benefit can be provided by several different features.