Troubleshooters.Com and T.C Linux Library and Nullmailer Docs Present

Nullmailer Landmine Map

Copyright (C) 2013 by Steve Litt, All rights reserved. Material provided as-is, now warranty, use at your own risk. 



This page is a partial repair manual for Nullmailer, which is software to send email without a full blown MTA. It's written by Steve Litt, and Steve Litt is not a developer in the Nullmailer project, so there will be errors and oversimplifications. This is a partial repair manual, not an authoritative text. This page may have errors -- in fact it probably does. DO NOT trust what you see on this page without verifying it for yourself. I am not responsible for any damage or injury caused by your use of this document, or caused by errors and/or omissions in this document. If that's not acceptable to you, you may not use this document. By using this document you are accepting this disclaimer.

The techniques in this document could be used to send spam. Don't do it. You could find yourself fined, prosecuted, and you'll certainly find yourself ostracized. If the people aren't your existing customers, don't do it.

Nullmailer + This Doc = Success

If you're trying to deal with Nullmailer and getting more frustrated by the minute, this Nullmailer Landmine Map is just what you need to get it running in style. Nullmailer is horribly underdocumented, and its error messages are legendarily unhelpful, but the fact is, when it works, it works very well. This document will help you see inside the black box so you can work intelligently with Nullmailer.

I'd highly suggest you read this entire page before starting your troubleshooting. You don't need to memorize and understand everything, but at least be familiar with what's available for Nullmailer troubleshooting. Then, as your troubleshoot proceeds, you can reread parts of this document in detail.

One other thing. You might ask yourself why not use one of the many alternatives. My answer is simple enough: all the alternatives are just as black-boxy and just as under-documented, and unlike Nullmailer, I couldn't even get those to work. SMTP pushers like Nullmailer are like cell phone companies and email clients: They all suck. Nullmailer just happens to suck less.

Nullmailer Mental Model

Here's a Mental Model of Nullmailer:

Mental Model diagram of

So here's what happens when you call nullmail-inject on a crude email file you've created...

The nullmail-inject program re-formats the file to an RFC822 email file, based on config info it gets from several config files. Note that it expects those config files in a certain place, and if it doesn't find them there, you'll get the dreaded "Could not load the config" message. Once nullmail-inject has RFC822 formatted the file, it spawns nullmailer-queue. nullmailer-queue simply places the file mail queue where the nullmailer-send program expects it, and then writes to the trigger pipe to wake up the perpetually running nullmailer-send program.

If you've ever written code for money, you're familiar with the paradigm of one program putting files in a directory, and a constantly spinning other program processing those files. It's a way that clients can submit files without needing to wait for them to be processed. It's an old paradigm, and a good one. Nullmailer works like this, and the constantly spinning program is called nullmailer-send. I'm going to describe the process for a lightly used system where the trigger wakes up nullmailer-send, but the case in which the queue is often full is similar -- the time to send all the mail exceeds the configured time between queue scans, and it just sends continuously. If you get it running either of these ways you'll likely get it running the other way too.

So nullmailer-send is sitting there sleeping, and nullmailer-queue submits a file to the queue and informs nullmailer-send via the trigger pipe. At this point, nullmailer-send goes through the queue directory, parsing one file after another. Presumably on a lightly used system there's only one file. For each file, nullmailer-send consults the remotes file, fills in the blanks, and takes the appropriate actions. By the way, if it can't find the remotes file, or if the file is too badly in error, you'll get the beloved "Could not load the config" error.

Before continuing this narrative, let's take a look at a typical remotes entry: smtp --port=465 --pass=mypassword --ssl
The first item is the remote Mail Transfer Agent, in this case running SMTP protocol. See that second item, smtp? Besides being a protocol name, is the name of the executable in the libexec/nullmailer directory that is spawned by nullmailer-send. That's obviously a security problem, but less obviously it's a great troubleshooting opportunity.

Troubleshooting Tools for Nullmailer

It's an idea. A method. It's been around since the hunter-gatherers settled down to farming, and had to repair their plows. It's a simple idea:

If the machine or system under repair seems like a black box, you need a tool to see inside it.

This article discusses tools you can use to "see inside" Nullmailer and get it up and running more quickly.


The first tool you need is the ps command.
slitt@mydesk:~$ ps ax | grep nullmail
6167 pts/15 S+ 0:00 nullmailer-send
6223 pts/9 S+ 0:00 grep --color=auto nullmail
If you were to run that command while libexec/nullmailer/smtp was sending something, you'd get an additional line, for the smtp executable, showing all its command line args including your email password, which is the major source of Nullmailer's insecurity on multi-user systems. Anyway, if nullmailer-send isn't running, you can't possibly expect Nullmailer to work.

Injection Shellscript

This is a shellscript to send an email at will. It puts the time into the email so that you can recognize which emails got through and which didn't. This script relieves you of the hassle of making an email file and manually calling nullmailer-inject with the exact right arguments. This script's simplicity eliminates the silly mistakes you might make. In troubleshooting, one silly mistake that causes you to assign the root cause to the wrong area can cost you hours (via "The Big Mistake").

Here's the injection shellscript:


youremail=$sender                       ##### $youremail: FEEL FREE TO MAKE IT SOMETHING ELSE
                                        #####     IF YOU HAVE A SECOND EMAIL ADDRESS    ##### $friendemail: AN EMAIL ADDRESS TO CC

yourname="Yourfirst Yourlast"           ##### CHANGE TO A NAME SUITABLE FOR TESTING
friendname="Friendfirst Friendlast"     ##### CHANGE TO A NAME SUITABLE FOR TESTING

echo "Subject: Nullmailer test at $dtime" > $emailfile
echo "From: $yourname  $ltlt$sender$gtgt" >> $emailfile
echo "To: $yourname  $ltlt$youremail$gtgt" >> $emailfile
echo "Cc: $friendname $ltlt$friendemail$gtgt" >> $emailfile
echo "" >> $emailfile
echo "Sent at $dtime" >> $emailfile
echo "" >> $emailfile
echo "$yourname was here" >> $emailfile
echo "and now is gone" >> $emailfile
echo "but left his name" >> $emailfile
echo "to carry on." >> $emailfile
echo "" >> $emailfile
echo "This is a second paragraph thats kinda long, really really long, so long that I truly hope that it does the right thing and wraps." >> $emailfile
echo "" >> $emailfile
echo "Sincerely" >> $emailfile
echo "$yourname" >> $emailfile

cat $emailfile | nullmailer-inject -h

To use this, simply change the variables at the top to represent your situation, and run it. If you have a working Nullmailer system and your nullmailer-send is currently running, this will send a timestamped email to $youremail and $friendemail.

nullmailer-send output

When you run nullmailer-send you can get somewhat of an idea what's going on. Here's what it looks like when you start nullmailer-send in a functional Nullmailer system with no pending emails queued:

root@mydesk:/usr/local/libexec/nullmailer# nullmailer-send
Rescanning queue.

Here's what nullmailer-send looks like when you successfully send a message:

Trigger pulled.
Rescanning queue.
Starting delivery, 1 message(s) in queue.
Starting delivery: protocol: smtp host: file: 1361656907.7218
smtp: Succeeded: 250 OK id=1U9NAB-0003qT-5J
Sent file.
Delivery complete, 0 message(s) remain.

So what you do is you view nullmail-send in one (root) terminal, while in another visible terminal your run

Sometimes you see one of these error messages in the running nullmailer-send:

What all three of the preceding error messages have in common is they don't name the explicit file they couldn't exec, load or open. That information would make troubleshooting Nullmailer ten times easier. So, if you see one or more of those error messages, the good news is you know approximately what's going wrong, but the bad news is you have no idea of the location of the file involved.

If you get the "Could not open trigger file: Permission denied" error, you might be in luck. That could be as simple as running as the wrong user. Try running nullmailer-send as root, and if that doesn't work as user nullmail.

libexec/nullmailer/smtp Substitute (

The nullmailer-send program uses the second item in a etc/nullmailer/remotes file as the name of a program to run within the libexec/nullmailer directory. Although this could pose a security violation, it also presents a great troubleshooting opportunity, because you can drop in a simple shellscript to see everything libexec/nullmailer/smtp would receive from stdin, from environment variables, and from its command line argument. This shellscript is a spectacular troubleshooting tool. Here it is, place it in the libexec/nullmailer directory and give it the same ownership and permissions as the smtp file in that directory, change the second item in remotes to, rerun nullmailer-send, and run your email injection script:

echo ============= BEGIN $0 ===================
let argno=0
let argc=$#
echo Stdin content follows:
perl -n  -e 'print $_'
Environment vars follow:
echo Args follow, there are $argc args:
echo Arg0 is $0
while test $argno -lt $argc; do
        echo Arg$argno is \>$1\<
        let argno+=1
#echo -n Type in return value: 0 to 99:
#read rtrn
let rtrn=1
echo =============  END  $progname ===================
exit $rtrn

Now run it with this in remotes: --port=465 --pass=mypassword --ssl


Port 465 is the typical port for SMTP over SSL. The typical port for non-SSL is 25, but all of this varies from ISP to ISP, so you need to check the parameters in your own ISP's documentation, or if they don't have any, through their tech support.

============= BEGIN /usr/local/libexec/nullmailer/ ===================
Stdin content follows:

Received: (nullmailer pid 3882 invoked by uid 1000);
Sat, 23 Feb 2013 07:35:32 -0000
Subject: Nullmailer test at Sat Feb 23 02:35:32 EST 2013
From: Steve Litt <>
To: Steve Litt <>
Cc: Pineboard <>
Date: Sat, 23 Feb 2013 02:35:32 -0500
Message-Id: <>

Sent at Sat Feb 23 02:35:32 EST 2013

Steve Litt was here
and now is gone
but left his name
to carry on.

This is a second paragraph thats kinda long, really really long, so long that I truly hope that it does the right thing and wraps.

Steve Litt

Environment vars follow:
LESSOPEN=| /usr/bin/lesspipe %s
LESSCLOSE=/usr/bin/lesspipe %s %s

Args follow, there are 5 args:
Arg0 is /usr/local/libexec/nullmailer/
Arg1 is >--port=465<
Arg2 is ><
Arg3 is >--pass=mypassword<
Arg4 is >--ssl<
Arg5 is ><
============= END /usr/local/libexec/nullmailer/ ===================
Sending failed: Unspecified temporary error
Delivery complete, 2 message(s) remain.

The error message at the end of the preceding printout is caused by the mail not actually being sent by the test shellscript. That error message might be suppressed by the test shellscript returning a specific value, but that might actually be misleading.

Pretty cool, huh? You can test on its own in your libexec/nullmailer/ directory, verify it works, then run nullmailer-send and see if you get a printout. If it doesn't print out when you send a test message, that's a pretty good indicater that you've got the wrong directory (Sending failed: Could not exec program), or perhaps the wrong permissions. Find out what's going on. This is one way to peer inside the black box that is Nullmailer.

But Wait, There's More!

Notice that when you switched remotes to execute instead of smtp, you got a printout of everything that came in from Stdin, from environment variables, and from command line arguments. You know what that means, don't you? It means you know every bit of info that was supposed to be passed from nullmailer-send to libexec/nullmailer/smtp. In turn, that means you can set up a test direct to libexec/nullmailer/smtp, eliminating the need for nullmailer-inject, nullmailer-queue, nullmailer-send, and the remotes file.

Basically, what you do is copy the Stdin stuff to a file, copy the arguments to a shellscript that calls libexec/nullmailer/smtp with those arguments, and within the shellscript export any relevant environment variables (which most likely would be $HELOHOST). So in my test file, if the Stdin file were stdin.txt, the shellscript, which should be run as the user normally running nullmailer-send, should look like this:  /usr/local/libexec/nullmailer/smtp --port=465 --pass=mypassword --ssl < stdin.txt

Now that you're dealing directly with libexec/nullmailer/smtp, and only with libexec/nullmailer/smtp, you can change one factor at a time, seeing how things change, until you get it working, without worrying about all sorts of extraneous factors. Note that if it appears to hang, chances are it's waiting for the end of Stdin. Hit Ctrl+D (end of file) and see if that ends the "hang". If so, you probably just forgot to redirect in the stdin.txt file.


Be aware that the stdin.txt file is static, meaning it will have the same timestamp every time, so if you want to differentiate tries, you'll need to edit it every time. I don't think that's too important because smtp tells you whether it succeeds or fails, and if it succeeds, you'll probably receive the email. If not, you can troubleshoot further, maybe modifying stdin.txt each time.

Locate Command

One of the toughest things to do is figure out where the Nullmailer system expects to see its configuration files and its executable files. One would think that the executables would tell this either with an error message or with their --help option.

But noooooooooo! You have to figure it out yourself, pretty much by trial and error. With perhaps hundreds of subdirectories on your computer, you have to trial and error it. The good news is, you can probably greatly narrow the search. Do this:

sudo updatedb
locate nullmailer | grep "/etc"
locate nullmailer | grep "/bin"
locate nullmailer | grep "/sbin"
locate nullmailer | grep "/libexec"

Those likely will give you a pretty good idea where to look for Nullmailer's config and executables. If you've had other Nullmailer installations on this computer, you could get several. And conceivably, if some package maintainer thought it was a good idea to put these things in paths not including the word "nullmailer", the preceding might yield wrong or no information. But chances are, they'll yield the correct directories, and not too many others.

From there, it's just a matter of changing remotes to run instead of smtp, and see whether the change takes place. And if it does, see the output of A little trial and error should positively indicate the correct directories. And of course, if worse comes to worse, in a candidate directory you can rename a file and see whether the symptom changes.

Troubleshooting Ladder

When troubleshooting a problem on a complicated system, especially a system where things depend on each other, many times the best way to start is with a canned diagnostic test ladder of increasing difficulty. That's the basis of Samba's testparm file, for instance. I'd recommend that for Nullmailer too. If you already have all your tools, this ladder should take you less than a half hour.
  1. ping, or whatever the URL of the SMTP server you're trying to send to. If you have no connectivity, nothing else will work. If ping fails, make sure it's not a firewall problem on your end or theirs. Hint: If you can't ping anything on the Internet, that's a good indication that you have a connectivity or firewall problem on your end. Also, try the traceroute and nmap. If you can find no evidence of the far end, this might be a good time to call tech support at your ISP.
  2. Verify user and group nullmail by grepping against /etc/passwd and /etc/group as root. Nullmailer specifications say that this user and group is necessary to run Nullmailer. If they're not there, use adduser to create them.
  3. Find Nullmailer directories
    1. sudo updatedb or su -c updatedb to update the locate database
    2. locate -r /smtp$ | grep libexec to find the libexec(s)
    3. locate -r /nullmailer-send$ to find Nullmailer's sbin directory(s)
    4. locate -r /nullmailer-inject$ to find Nullmailer's bin directory(s)
    5. locate -r /remotes$ to find Nullmailer's etc directory(s)
    6. For each of the preceding directory categories, find the legitimate directory used by the currently installed Nullmailer version via experimentation, such as renaming a needed file and seeing if the symptom changes. If none of the listed directories respond to a file rename, it's likely the correct directories don't exist.
  4. Verify that nullmailer-send is running when you send a message. If it's not, the message can't possibly get sent.
  5. Verify that nullmailer-send is running as either root or user nullmail. If it's not, there will probably be problems.
  6. Verify that the SMTP login info you're using: name, password, SSL status, port, protocol and the like work in an email program. If you've been regularly sending email from an email client on your computer, this info is what you need to use in your remotes file.
  7. If you're trying to use SSL, verify that your libexec/nullmailer/smtp is SSL capable with this command:
    libexec/nullmailer/smtp --help
    If it doesn't list an argument called --ssl, this program cannot possibly do SSL and will throw an error message if you put --ssl in remotes. If this command doesn't show --ssl as a possible argument, you'll need to recompile Nullmailer 1.10 or later using the --enable-tls argument to ./configure, and then, as the make repeatedly aborting because of dependencies, you'll need to install packages to fulfill those dependencies. You're almost certainly going to need to install  the gnutls and gnutls dev packages, although of course they could be called something else.
  8. If the preceding test indicated SSL support, verify that your line in remotes contains the --ssl flag, or else it can't possibly do SSL.
  9. Use

Diagnostic Prints

If worst comes to worst and you can't do it any other way, you can insert your own print messages. Perhaps something like this:
fprintf(stderr, "Config file is: >%s<\n", fullname.c_str());
If you were to put that in lib/config_readlist, function config_readlist() after the construction of fullname, you would know exactly what config file you were reading, and have a heck of a lot easier time interpreting a later error message.

Yes, I know, this is a C++ file and I should have used the cerr standard stream. If C++ standard streams float your boat, by all means use them. Personally, I find the printf family much easier. One nice thing about C++ is you can use C constructs when you want to.


Before changing source code, always back up the original, so when you're done you can "put it back." If you ever seek help from others, especially the software's author, it's only fair that you're dealing with the original product rather than your own hack.

By inserting diagnostic prints inside the source code and recompiling, you can view variables and progress. If you can run it inside a debugger, do that instead: it is quicker and easier.


Especially when dealing with a program like Nullmailer, with non-specific error messages, sparse commenting, and hard-to-trace Volleyball Code, grep is your friend. Want to know which code threw an error message? Do this:
grep -irl "The specific error message" | grep -v \.o$
Trying to trace a piece of data, object or function? Do this:
grep -irl "cli_options" | grep -v \.o$
Grep makes the impossible possible, and the possible easy.

Fixing the Security Problem

Nullmailer's security problem is that your SMTP server's password is delivered to the smtp executable on the commaned line, meaning it's visible to anyone with local or remote access to the computer, via the ps command. Changing it to an environment variable wouldn't be any better, because the environment variable would be available in the executable's pid directory's environ file.

What you need to do is have the password read from a file readable by the user running the smtp executable, but no other user. Probably its group should be root. So here's how it would work, assuming that user nullmail is running the smtp executable...

Remember, the remotes file looks like this: smtp --port=465 --pass=mypassword --ssl

In this case, you would change the literal mypassword to the filename. For instance:


Where both the directory pass and the file within it are owned by user nullmail, group root and chmod 700 and 600, respectively. Then the smtp executable would read the file and use it for the password, but those looking at a ps command would only know the filename to a file they could not read.

Unfortunately, due to the way Nullmailer is written (Volleyball Code)), I couldn't find where the password gets set inside the executable, so I was unable to make the change in the amount of time I'd allocated to do so. Keep in mind that only I use my system, so for me it's not much of a security problem.

If you want to make the change, I'd recommend setting a breakpoint at the password to see when it gets set, and also find out where the remotes file gets read. You can then just put the password filename as a separate variable or object property somewhere, and then use it to do the read where the password gets set.


Like so many systems and machines before it, Nullmailer was created with little thought toward repairability. This means you'll need to use special techniques to repair Nullmailer problems. These special techniques include:

Back to Troubleshooters.Com * Back to Linux Library * Back to Nullmailer Docs