Nullmailer Landmine Map
Copyright (C) 2013 by Steve Litt, All
rights reserved. Material provided as-is, now warranty, use at your own
This page is a partial repair manual for Nullmailer, which is software to send
email without a full blown MTA. It's written by Steve Litt, and Steve Litt
is not a developer in the Nullmailer project, so there will be errors and
oversimplifications. This is a partial repair manual, not an authoritative
text. This page may have errors -- in fact it probably does. DO NOT trust
what you see on this page without verifying it for yourself. I am not
responsible for any damage or injury caused by your use of this document, or
caused by errors and/or omissions in this document. If that's not acceptable
to you, you may not use this document. By using this document you are
accepting this disclaimer.
The techniques in this document could be used to send spam. Don't do it. You
could find yourself fined, prosecuted, and you'll certainly find yourself
ostracized. If the people aren't your existing customers, don't do it.
Nullmailer + This Doc =
If you're trying to deal with Nullmailer and getting more frustrated by the
minute, this Nullmailer Landmine Map is just what you need to get it running
in style. Nullmailer is horribly underdocumented, and its error messages are
legendarily unhelpful, but the fact is, when it works, it works very well.
This document will help you see inside the black box so you can work
intelligently with Nullmailer.
I'd highly suggest you read this entire page before starting your
troubleshooting. You don't need to memorize and understand everything, but
at least be familiar with what's available for Nullmailer troubleshooting. Then, as
your troubleshoot proceeds, you can reread parts of this document in detail.
One other thing. You might ask yourself why not use one of the many
alternatives. My answer is simple enough: all the alternatives are just as
black-boxy and just as under-documented, and unlike Nullmailer, I couldn't
even get those to work. SMTP pushers like Nullmailer are like cell phone
companies and email clients: They all suck. Nullmailer just happens to suck
Nullmailer Mental Model
Here's a Mental Model of Nullmailer:
So here's what happens when you call nullmail-inject
on a crude email file you've created...
The nullmail-inject program
re-formats the file to an RFC822 email file, based on config info it gets
from several config files. Note that it expects those config files in a
certain place, and if it doesn't find them there, you'll get the dreaded
"Could not load the config" message. Once nullmail-inject has RFC822 formatted the file, it
spawns nullmailer-queue. nullmailer-queue simply places the
file mail queue where the nullmailer-send
program expects it, and then writes to the trigger pipe to wake up the
perpetually running nullmailer-send
If you've ever written code for money, you're familiar with the paradigm of
one program putting files in a directory, and a constantly spinning other
program processing those files. It's a way that clients can submit files
without needing to wait for them to be processed. It's an old paradigm, and
a good one. Nullmailer works like this, and the constantly spinning program
is called nullmailer-send. I'm
going to describe the process for a lightly used system where the trigger
wakes up nullmailer-send, but
the case in which the queue is often full is similar -- the time to send all
the mail exceeds the configured time between queue scans, and it just sends
continuously. If you get it running either of these ways you'll likely get it
running the other way too.
So nullmailer-send is sitting
there sleeping, and nullmailer-queue
submits a file to the queue and informs nullmailer-send via the trigger pipe. At this point, nullmailer-send goes through the
queue directory, parsing one file after another. Presumably on a lightly
used system there's only one file. For each file, nullmailer-send consults the remotes file, fills in the blanks, and takes the
appropriate actions. By the way, if it can't find the remotes file, or if the file is too
badly in error, you'll get the beloved "Could not load the config" error.
Before continuing this narrative, let's take a look at a typical remotes entry:
smtp.myisp.com smtp --port=465 --firstname.lastname@example.org --pass=mypassword --ssl
The first item is the remote Mail Transfer Agent, in this case running SMTP
protocol. See that second item, smtp?
Besides being a protocol name, is the name of the executable in the libexec/nullmailer directory that
is spawned by nullmailer-send.
That's obviously a security problem, but less obviously it's a great
Tools for Nullmailer
It's an idea. A method. It's been around since the hunter-gatherers
settled down to farming, and had to repair their plows. It's a simple
|If the machine or system under repair seems like a black
box, you need a tool to see inside it.
This article discusses tools you can use to "see inside" Nullmailer and get
it up and running more quickly.
The first tool you need is the ps
slitt@mydesk:~$ ps ax | grep nullmail
If you were to run that command while libexec/nullmailer/smtp
was sending something, you'd get an additional line, for the smtp executable, showing all its
command line args including your email password, which is the major source
of Nullmailer's insecurity on multi-user systems. Anyway, if nullmailer-send isn't running, you
can't possibly expect Nullmailer to work.
6167 pts/15 S+ 0:00 nullmailer-send
6223 pts/9 S+ 0:00 grep --color=auto nullmail
This is a shellscript to send an email at will. It puts the time into the
email so that you can recognize which emails got through and which didn't.
This script relieves you of the hassle of making an email file
and manually calling nullmailer-inject
with the exact right arguments. This script's simplicity eliminates the
silly mistakes you might make. In troubleshooting, one silly mistake that
causes you to assign the root cause to the wrong area can cost you hours (via "The Big Mistake").
Here's the injection shellscript:
# START OF USER
##### $sender: MUST BE YOUR EMAIL AT THE
##### $youremail: FEEL FREE TO MAKE IT
##### IF YOU
HAVE A SECOND EMAIL ADDRESS
$friendemail: AN EMAIL ADDRESS TO CC
##### CHANGE TO A NAME SUITABLE FOR TESTING
Friendlast" ##### CHANGE TO A NAME
SUITABLE FOR TESTING
# END OF
USER CHANGABLE VARIABLES
Nullmailer test at $dtime" > $emailfile
$yourname $ltlt$sender$gtgt" >> $emailfile
$yourname $ltlt$youremail$gtgt" >> $emailfile
echo "Cc: $friendname
$ltlt$friendemail$gtgt" >> $emailfile
echo "" >>
echo "Sent at $dtime"
echo "" >>
echo "$yourname was
here" >> $emailfile
echo "and now is
gone" >> $emailfile
echo "but left his
name" >> $emailfile
echo "to carry on."
echo "" >>
echo "This is a
second paragraph thats kinda long, really really long, so long
that I truly hope that it does the right thing and wraps."
echo "" >>
cat $emailfile |
To use this, simply change the variables at the top to represent your
situation, and run it. If you have a working Nullmailer system and your nullmailer-send is currently
running, this will send a timestamped email to $youremail and $friendemail.
When you run nullmailer-send
you can get somewhat of an idea what's going on. Here's what it looks like
when you start nullmailer-send
in a functional Nullmailer system with no pending emails queued:
Here's what nullmailer-send looks like when you successfully send
Starting delivery, 1 message(s) in queue.
Starting delivery: protocol: smtp host: mail.a3b3.com file:
smtp: Succeeded: 250 OK id=1U9NAB-0003qT-5J
Delivery complete, 0 message(s) remain.
So what you do is you view nullmail-send in one (root)
terminal, while in another visible terminal your run
Sometimes you see one of these error messages in the running nullmailer-send:
What all three of the preceding error messages have in common is they don't
name the explicit file they couldn't exec, load or open. That
information would make troubleshooting Nullmailer ten times easier. So, if
you see one or more of those error messages, the good news is you know
approximately what's going wrong, but the bad news is you have no idea of
the location of the file involved.
- Sending failed: Could not exec program
- Could not load the config
- Could not open trigger file: Permission denied
If you get the "Could not open trigger file: Permission denied" error, you
might be in luck. That could be as simple as running as the wrong user. Try
running nullmailer-send as
root, and if that doesn't work as user nullmail.
The nullmailer-send program
uses the second item in a etc/nullmailer/remotes
file as the name of a program to run within the libexec/nullmailer directory. Although this could pose
a security violation, it also presents a great troubleshooting opportunity,
because you can drop in a simple shellscript to see everything libexec/nullmailer/smtp would
receive from stdin, from environment variables, and from its command line
argument. This shellscript is a spectacular troubleshooting tool. Here it
is, place it in the libexec/nullmailer
directory and give it the same ownership and permissions as the smtp file in that directory, change
the second item in remotes to
test.sh:, rerun nullmailer-send, and run your email
BEGIN $0 ===================
echo Stdin content
perl -n -e
Environment vars follow:
echo Args follow,
there are $argc args:
echo Arg0 is $0
while test $argno -lt
echo Arg$argno is \>$1\<
#echo -n Type in
return value: 0 to 99:
============= END $progname ===================
Now run it with this in remotes:
smtp.myisp.com test.sh --port=465 --email@example.com --pass=mypassword --ssl
Port 465 is the typical port for SMTP over SSL. The typical port for non-SSL
is 25, but all of this varies from ISP to ISP, so you need to check the
parameters in your own ISP's documentation, or if they don't have
any, through their tech support.
pid 3882 invoked by uid 1000);
Sat, 23 Feb 2013
test at Sat Feb 23 02:35:32 EST 2013
From: Steve Litt
To: Steve Litt
Date: Sat, 23 Feb
2013 02:35:32 -0500
Sent at Sat Feb 23
02:35:32 EST 2013
Steve Litt was here
and now is gone
but left his name
to carry on.
This is a second
paragraph thats kinda long, really really long, so long that I
truly hope that it does the right thing and wraps.
Environment vars follow:
LESSOPEN=| /usr/bin/lesspipe %s
LESSCLOSE=/usr/bin/lesspipe %s %s
Args follow, there
are 5 args:
Arg4 is >--ssl<
Unspecified temporary error
Delivery complete, 2
error message at the end of the
preceding printout is caused by the mail not actually being sent by the test
shellscript. That error message might be suppressed by the test shellscript
returning a specific value, but that might actually be
Pretty cool, huh? You can test test.sh
on its own in your libexec/nullmailer/
directory, verify it works, then run nullmailer-send
and see if you get a printout. If it doesn't print out when you send a test
message, that's a pretty good indicater that you've got the wrong directory
(Sending failed: Could not exec program), or perhaps the wrong permissions.
Find out what's going on. This is one way to peer inside the black box that
But Wait, There's More!
Notice that when you switched remotes
to execute test.sh instead of
smtp, you got a printout of
everything that came in from Stdin, from environment variables, and from
command line arguments. You know what that means, don't you? It means you
know every bit of info that was supposed to be passed from nullmailer-send to libexec/nullmailer/smtp. In turn,
that means you can set up a test direct to libexec/nullmailer/smtp, eliminating the need for nullmailer-inject, nullmailer-queue,
nullmailer-send, and the remotes
Basically, what you do is copy the Stdin stuff to a file, copy the arguments
to a shellscript that calls libexec/nullmailer/smtp
with those arguments, and within the shellscript export any relevant environment
variables (which most likely would be $HELOHOST). So in my test file,
if the Stdin file were stdin.txt,
the shellscript, which should be run as the user normally running nullmailer-send, should look like
/usr/local/libexec/nullmailer/smtp --port=465 --firstname.lastname@example.org
--pass=mypassword --ssl smtp.myisp.com
Now that you're dealing directly with libexec/nullmailer/smtp,
and only with libexec/nullmailer/smtp,
you can change one factor at a time, seeing how things change, until you get
it working, without worrying about all sorts of extraneous factors. Note
that if it appears to hang, chances are it's waiting for the end of Stdin.
Hit Ctrl+D (end of file) and see if that ends the "hang". If so, you
probably just forgot to redirect in the stdin.txt file.
Be aware that the stdin.txt
file is static, meaning it will have the same timestamp every time,
so if you want to differentiate tries, you'll need to edit it every
time. I don't think that's too important because smtp tells you whether it
fails, and if it succeeds, you'll probably receive the email. If
not, you can troubleshoot further, maybe modifying stdin.txt each time.
One of the toughest things to do is figure out where the Nullmailer
system expects to see its configuration files and its executable files.
One would think that the executables would tell this either with an error
message or with their --help
But noooooooooo! You have
to figure it out yourself, pretty much by trial and error. With perhaps
hundreds of subdirectories on your computer, you have to trial and error
it. The good news is, you can probably greatly narrow the search. Do this:
locate nullmailer | grep
locate nullmailer | grep
locate nullmailer | grep
locate nullmailer | grep "/libexec"
Those likely will give you a pretty good idea where to look for
Nullmailer's config and executables. If you've had other Nullmailer
installations on this computer, you could get several. And conceivably, if
some package maintainer thought it was a good idea to put these things in
paths not including the word "nullmailer", the preceding might yield wrong
or no information. But chances are, they'll yield the correct directories,
and not too many others.
From there, it's just a matter of changing remotes to run test.sh
instead of smtp, and see
whether the change takes place. And if it does, see the output of test.sh. A little trial and error
should positively indicate the correct directories. And of course, if
worse comes to worse, in a candidate directory you can rename a file and
see whether the symptom changes.
When troubleshooting a problem on a complicated system, especially a system
where things depend on each other, many times the best way to start is with
a canned diagnostic test ladder of increasing difficulty. That's the basis
of Samba's testparm file, for instance. I'd recommend that for
Nullmailer too. If you already have all your tools, this ladder should take
you less than a half hour.
- ping smtp.myisp.com, or
whatever the URL of the SMTP server you're trying to send to. If you
have no connectivity, nothing else will work. If ping fails, make sure it's not
a firewall problem on your end or theirs. Hint: If you can't ping
anything on the Internet, that's a good indication that you have a
connectivity or firewall problem on your end. Also, try the traceroute and nmap. If you can find no
evidence of the far end, this might be a good time to call tech support
at your ISP.
- Verify user and group nullmail
by grepping against /etc/passwd
and /etc/group as root.
Nullmailer specifications say that this user and group is necessary to
run Nullmailer. If they're not there, use adduser to create them.
- Find Nullmailer directories
- sudo updatedb or su -c updatedb to update the
- locate -r /smtp$ | grep
libexec to find the libexec(s)
- locate -r /nullmailer-send$
to find Nullmailer's sbin
- locate -r /nullmailer-inject$
to find Nullmailer's bin
- locate -r /remotes$ to
find Nullmailer's etc
- For each of the preceding directory categories, find the legitimate
directory used by the currently installed Nullmailer version via
experimentation, such as renaming a needed file and seeing if the symptom
changes. If none of the listed directories respond to a file rename,
it's likely the correct directories don't exist.
- Verify that nullmailer-send
is running when you send a message. If it's not, the message can't
possibly get sent.
- Verify that nullmailer-send
is running as either root or user nullmail. If it's not, there will
probably be problems.
- Verify that the SMTP login info you're using: name, password, SSL status,
port, protocol and the like work in an email program. If you've been
regularly sending email from an email client on your computer, this info
is what you need to use in your remotes
- If you're trying to use SSL, verify that your libexec/nullmailer/smtp is SSL
capable with this command:
If it doesn't list an argument called --ssl,
this program cannot possibly do SSL and will throw an error message if
you put --ssl in remotes. If this command
doesn't show --ssl as a
possible argument, you'll need to recompile Nullmailer 1.10 or later
using the --enable-tls
argument to ./configure,
and then, as the make
repeatedly aborting because of dependencies, you'll need to install
packages to fulfill those dependencies. You're almost certainly going to
need to install the gnutls and gnutls dev
packages, although of course they could be called something else.
- If the preceding test indicated SSL support, verify that your line in
remotes contains the --ssl flag, or else it can't
possibly do SSL.
- Use test.sh.
If worst comes to worst and you can't do it any other way, you can insert
your own print messages. Perhaps something like this:
fprintf(stderr, "Config file is: >%s<\n", fullname.c_str());
If you were to put that in lib/config_readlist,
after the construction of fullname,
you would know exactly what config file you were reading, and have a heck of
a lot easier time interpreting a later error message.
Yes, I know, this is a C++ file and I should have used the cerr standard stream. If
C++ standard streams float your boat, by all means use them.
Personally, I find the printf
family much easier. One nice thing about C++ is you can use C
constructs when you want to.
Before changing source code, always back up the original, so when
you're done you can "put it back." If you ever seek help from
others, especially the software's author, it's only fair that you're
dealing with the original product rather than your own hack.
By inserting diagnostic prints inside the source code and recompiling, you can
view variables and progress. If you can run it inside a debugger, do that instead:
it is quicker and easier.
Especially when dealing with a program like Nullmailer, with non-specific
error messages, sparse commenting, and hard-to-trace Volleyball Code, grep is your friend. Want to know
which code threw an error message? Do this:
grep -irl "The specific error message" | grep -v \.o$
Trying to trace a piece of data, object or function? Do this:
grep -irl "cli_options" | grep -v \.o$
Grep makes the impossible possible, and the possible easy.
Fixing the Security Problem
Nullmailer's security problem is that your SMTP server's password is
delivered to the smtp executable on the commaned line, meaning it's
visible to anyone with local or remote access to the computer, via the
ps command. Changing it to an environment variable wouldn't be any
better, because the environment variable would be available in the
executable's pid directory's environ file.
What you need to do is have the password read from a file readable by the
user running the smtp executable, but no other user. Probably its
group should be root. So here's how it would work, assuming that
user nullmail is running the smtp executable...
Remember, the remotes file looks like this:
smtp.myisp.com smtp --port=465 --email@example.com --pass=mypassword --ssl
In this case, you would change the literal mypassword to the
filename. For instance:
Where both the directory pass and the file within it are owned
by user nullmail, group root and chmod 700 and 600,
respectively. Then the smtp executable would read the file and use
it for the password, but those looking at a ps command would only
know the filename to a file they could not read.
Unfortunately, due to the way Nullmailer is written (Volleyball
Code)), I couldn't find where the password gets set inside the
executable, so I was unable to make the change in the amount of time I'd
allocated to do so. Keep in mind that only I use my system, so for me it's
not much of a security problem.
If you want to make the change, I'd recommend setting a breakpoint at the
password to see when it gets set, and also find out where the
remotes file gets read. You can then just put the password filename
as a separate variable or object property somewhere, and then use it to do
the read where the password gets set.
Like so many systems and machines before it, Nullmailer was created with little
thought toward repairability. This means you'll need to use special techniques
to repair Nullmailer problems. These special techniques include:
- Use the Mental Model in this magazine, perhaps with your own additions,
perhaps added by use of reverse engineering techniques
- Use tools to look at and strongarm Nullmailer components.
Some tools named in this document include:
- ps command
- Injection shellscript
- Viewing output of nullmailer-send
- libexec/nullmailer/smtp substitute (test.sh)
- Direct manipulation of the smtp command
- locate command
- Troubleshooting ladder
- Debugger and/or diagnostic prints
- The grep command
- Keep a cool head (The Attitude)