Troubleshooters.Com® and Email Tech Present:

Email Troubleshooting, Including IMAP, Dovecot, and Linux

Career Skills nobody else teaches


CONTENTS:

Introduction

Ever have trouble getting your Dovecot IMAP server to work? Ever have trouble just getting your email client to connect to somebody else's IMAP server (like Google Mail)? Or even to your own IMAP server that you control? There's a reason you had trouble: too many variables:

And those are when the IMAP server is correctly configured. When your IMAP server has problems, the variables are beyond count, but here are some of the usual suspects:

All these variables, and often your only test points are the complicated config options on an email client. No wonder imap's so hard to troubleshoot. This document gives you the testpoints to transform IMAP connectivity from a black box to a systematic narrowing of the root cause scope, which, after all, is the foundation of the Universal Troubleshooting Process. And in this documents you'll learn a few techniques to make IMAP setup and troubleshooting easier.

So kick back, relax, and read this document. If you're not scared to dig into technology, but you're sick of the hassles of IMAP repair, this document is for you.

Steve Litt is the author of the Universal Troubleshooting Process Courseware. Steve can be reached at his email address.

Note to Windows and Mac Users

This document was written using Linux commands, and tech edited on Linux computers. That being said, the principles involved in this document just might help you.

Steve Litt is the author of several books on Rapid Learning. Steve can be reached at his email address.

Ports and Protocols

Oh, heaven help me, I wish it weren't necessary to write this section. But believe me, it's necessary. Between interchangeable use of technical terms, different ways of doing things, and progress, the terminology of email communication is ambiguous almost to the point of uselessness, at least without the understanding this section brings. Further compounding the situation is the confusion of port numbers with protocols. This confusion has everything to do with custom, and nothing to do with technology.

What Are Ports?

Every IP address has 65535 ports for TCP, and 65535 for UDP, all numbered 1 to 65535. Port 0 is reserved and unusable.

Note

Because there are 65535 ports for TCP, and 65535 for UDP, one might be tempted to say there are 13170 ports. As a matter of custom, and for preventing confusion, in the real world, that's just not true. Among the well-known ports, the only cases I know of where the same port number is used both by a UDP service and a TCP service is where both services are part of one service. For instance, the IMAPS service uses both TCP port 993 and UDP port 993.

Each port is a "place" where a computer program can attach itself to communicate with the outside world. It listens on the port, and responds on the same port or another port. By "listens", I mean it sits there waiting for an outside computer program to query it over the network, after which it answers the query and converses with the outside program. When I say "converses", I mean each side sends and receives messages in a prescribed way so that each understands the other. That "prescribed way" is called a "protocol". As an example, customarily a web browser and a web server have their discussion on port 80, using the protocol called HTTP.

Technologically, any program on any computer can use any port for any purpose. There's no law of physics, nor any central Internet wide database, that forces the HTTP protocol to work only on port 80, and port 80 to carry only HTTP protocol. Instead, the association between the HTTP protocol and port 80 is a custom. A very well entrenched custom. If you wanted, you could set your ssh server to operate on port 80. The only problem is, it would conflict with any access too or from the outside, because the rest of the world thinks port 80 is HTTP protocol. Bad things would happen. But if all your communication were in your own house, you could set up every computer to use port 80 for ssh, and every computer to use port 22, the usual ssh port, for HTTP, and everything would work fine.

As an analogy, there's no law of physics, or even law of the state or federal legislature, specifying that when there's a line you get at the back of the line, instead of the middle or the front. Getting in the back of the line is nothing more or less than a custom. But everyone over 15 years old knows that when somebody gets in the middle or front of a line, or anywhere but the back, bad things often happen. So those of us not pursuing conflict get in the back of the line. If you wanted, within your own home, you could set up a custom where you get in the front of the line, and people are pulled off the back. That works fine within your house, but lousy outside. Just like switching the usage of port 80 and port 22 would work fine inside your house, but would cause conflict when communicating with those outside your house. So, in most cases, the most practical thing is to follow the society's custom, whether inside or outside your house.

If you want to see a list of the customary services and their ports, look at file /etc/services on any Linux, Unix or BSD computer. But remember, that's a list of customary services, not necessarily the services actually used by you, or by those you communicate with.

There's a magic number, 1024. Ports under this number are considered special "system ports" and handled a special way. If you wonder why most ports you ever hear about are lower than 1024, it's because commonly used protocols gain advantages by the special handling afforded ports under 1000. But of course, there are few remaining ports under 1000 that haven't, by custom, been assigned to a specific protocol.

Ports Are Considered Scarce

Every communication program needs a port. There are probably millions of communication programs, many using their own proprietary protocols. It would be easy to consume all 65535 ports. It would be especially easy to consume the system ports whose numbers are under 1024. This has led to some decisions to implement technologies that conserved ports, in some cases by having one port speak two different protocols. Some such decisions, such as STARTTLS, have created new problems even while fixing others.

My personal view, whether creating a server or operating a client, I like one port to handle one protocol, with that protocol being encrypted. I don't want my password going out in plain text to the Internet.

Protocols, a Ball of Confusion

Ask five people the meaning of "the cloud", and you'll get five different answers. Sure, they'll all sound kind of like "the stuff's on the Internet", but they'll be materially different. Ask five software developers the meaning of "semantic" in software development, and you'll get five different answers. Sure, all of them will sound like "author for meaning, not for appearance", but they'll be different. I try to stay away from words like that.

But when dealing with viewing/fetching emails, you can't stay away from such words. Because people throw around words like tls, ssl, STARTTLS and encrypted interchangeably. Not just dummies who download porn and watch Jerry Springer, but smart people who write very complex computer programs. The Dovecot IMAP server's config file has several such confusions. So does the document you're reading now.

The STARTTLS Dilemma

Isn't STARTTLS cool? One single port can accommodate both plain text and TLS (or SSL) communication. Sounds great, doesn't it? Consume only one port. If both computers speak the same SSL/TLS dialect, everything's secure, otherwise, you can still communicate, but in plain text. Cool!

Um, except some people, me for instance, would vastly prefer no communication at all rather than endure the possibility of insecure communication. I don't want snoopers reading my password with a simple sniffer. For me, the entry field in my email client shouldn't be labeled "STARTTLS", it should be labeled "do you feel lucky, punk?".

I think, but don't know for certain, that you can set many email clients so that they won't connect via plain text no matter what. But I don't know how to make that setting, and therefore, I don't know if I'm secure. Personally, I'd like to connect to a port that does one thing and does it well. That one thing: TLS. If the client and server can't speak TLS (or the older SSL) to each other, don't connect at all.

Customary Email Ports and Protocols

The following is a brief summary of traditional ports and protocols related to email:

Note about port 587

By custom and tradition, port 587 is a little more than just an alternate to port 25. Port 587 is typically meant to service authenticated email, rather than the no-identification wild-wild west that is typical of port 25. Port 587 is used a lot by ISPs offering a SMTP mechanism to their customers, and only their customers.

There is no law of physics or the universe specifying the preceding uses of the preceding ports. It's just a matter of custom. That being said, just like the custom of driving on the right of the road in the US, non-adherence to these customs in public could lead to some confusion.

My Preferences

Personally, I like all TLS all the time. I set my local Dovecot server to connect only via SSL/TLS, only on port 993 because that's the traditional port for IMAP over SSL/TLS. If someone tries to come in via plain text, or tries to come in on port 143, they're not getting in.

For the same reason, I always try to choose web hosts whose IMAP server has SSL/TLS, and nothing but SSL/TLS, on one specific port: Typically 993. In my Claws mail, I try never to enable STARTTLS, and if the IMAP server requires STARTTLS, I don't use that email account for anything more serious than mailing lists and online magazines.

Steve Litt is a the author of Escape From Kmail. Steve can be reached at his email address.

Anatomy of an IMAP Command

Before going on, it's important to know the anatomy of an IMAP command. Here it is:

cmdstring     command     arguments

cmdstring identifies a command, so the IMAP server can handle a voluminous stream of commands and send results back to the right places. When the IMAP server sends results back to you, it prefaces its OK message with that same string, so you know which of the many commands you've sent is being responding to. So in real life as opposed to manual keyboard input, each new command you send would need a new command string. But in the case of a human operating the openssl command, the human's too slow to get requests and replies mixed up, so as a practical matter, when manually typing in the commands, you can probably aa through ag and then back to aa, because aa through ag are easiest to type, and for the same reason, if you need more, ca through cg, da through dg, etc. If you need more commands than that, you might not want to do it manually.

command is the IMAP command. Examples of IMAP command include:

There are many more commands, but the preceding are all used later in this document. You might have noticed that I made these commands lower case, while many documenters list them as upper case. My experience with my copy of openssl hitting Gmail, Earthlink and my own Dovecot IMAP server tells me that these commands are case independent, and of course it's always easier and quicker to type lowercase. But if these commands aren't working for you, try them uppercase and see if it makes a difference.

arguments refers to the arguments of the command. For instance, the login command takes arguments username and password, so the command would look like the following:

mystring    login    myusername    mypass

Meanwhile, the examine command takes a mailbox as an argument, so, if my IMAP tree has a top level mailbox called "golug", the following sets that mailbox as the target of examination (but not modification):

mystring    examine    golug

By understanding IMAP commands, you're in a better position to troubleshoot.

Steve Litt is the author of The Key to Everyday Excellence. Steve can be reached at his email address.

Connecting Your Email Client to an IMAP Server

This section assumes that the IMAP server to which you're connecting is properly configured. Other sections of this document address fixing an improperly configured IMAP server. But for the purposes of this section, assume a properly configured IMAP server.

You've gotta love those ISPs. They give you perfect step-by-step email client configurations. As long as you're using MS Outlook. Or maybe a small list of other clients, usually Windows only, sometimes plus Thunderbird. Otherwise, you're left to deduce the IP address, port, protocol, and how your login name is constructed, from their click by click instructions. Don't you love when the instructions say "your client will tell you which protocols are available." Oh geez guys, just give me the numbers and names. And just in case you want life even more confusing, most of these ISPs keep their old instructions around forever, ready to be discovered in a web search. So even though their current IMAP server is imap.dumbisp.com, their old document tells you it's mail.dumbisp.com, which would have worked in 2007, but doesn't work today.

What all this means is, to a greater or lesser extent, you're going to need some trial and error to connect your client to their IMAP server. This sections goal is to reduce the trial and error from hours to minutes by giving you testpoints. As you learned earlier in this document, the following are some of the client-side variables when connecting to a known good IMAP server:

Details of how to isolate the preceding variables are given in later sections, but before reading them, you should understand the following:

This suggests a troubleshooting procedure based on the following hierarchy:

Email clients are emphatically not useful in diagnosis via the preceding hierarchy, because they lump all the variables at the same level. All the variables are a package deal with email clients, and the email client's error messages or logs are for the most part useless. So, if you have trouble connecting with an email client, instead of spending hours of trial and error trying to troubleshoot with that black box called an email client, you'll use the following tools:

Be sure to internalize this hierarchy for maximum troubleshooting productivity. Using the hierarchy and tools, you should be able to solve email client connection problems with a known good IMAP or POP server within minutes, not hours. Once you've used the tools to find URL, port, protocol, username and password, it's trivial to put those same values in your email client, knowing that these settings are correct.

Note

This document currently doesn't contain specific instructions for diagnosing POP3 connections, but all the same principles apply.

Please continue to the next section to get more specific.

Steve Litt is the author of The Key to Everyday Excellence. Steve can be reached at his email address.

Checking Domain or IP Address and Port

Apology

In this section and some following sections, in order to faithfully represent computer output while still remaining at least somewhat responsive and useable on smaller devices, a few lines of computer output have been shrunken quite a bit. These shrunken lines are no different from other lines of computer output: they're just longer and so I shrunk them.

A very few lines of computer output were too long, even after shrinking them to the maximum. So I split those lines and at the splits I put this graphical line continuation symbol: Graphical line continuation symbol

No user input was shrunk or split in this document.

Let's say that from a bunch of Internet searching you've managed to find a domain name and port and protocol for an ISP's IMAP server. Your first step is to check whether the domain name actually makes sense. Let's say you get imap.gmail.com at port 993. So check whether the domain makes sense, using the dig command:

dig imap.gmail.com

The preceding resolves to several IP addresses, so at least the domain name imap.gmail.com has some meaning. Note that if you'd instead used dig to look up stevelitt.dumbisp.com, dig would not have delivered an answer section.

Once you have a working domain name, you need a port to go with it. Use either telnet or openssl to check ports, depending on whether you think the port is SSL/TLS or open.

Danger Will Robinson

Never, never, NEVER put a password or any other secret information into a telnet session, because it will be visible to any snoops anywhere your packets travel. Use telnet only to check the most mundane things, like whether a port responds.

If, for some troubleshooting reason, you absolutely must put a password into telnet, then create a junk, throwaway email account with a throwaway password you'll never be tempted to use again for any valuable login ("mypass" is a good one), and troubleshoot with that. The second you're done with the troubleshooting, delete the temporary account.

For instance, let's say you want to use Gmail's ssl-enabled IMAP, and from what you've heard it's at imap.gmail.com at port 993. Since you suspect it's SSL, try it with the openssl command:

slitt@mydesk:~$ openssl s_client -connect imap.gmail.com:993 -quiet
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
* OK Gimap ready for requests from 68.204.228.160 ba7if2687949oab.268

Warning

In the server's response to the openssl connect, note these two lines:

verify error:num=20:unable to get local issuer certificate
verify return:0

This is caused by a your lack of a certificate on the client end, but it also means you as the client can't be sure who is on the server end. In this particular case, I know who gmail.com is, and I trust my DNS, so I worked without a certificate on the client end. Throughout the rest of this document I continued to work without a certificate. If you want more security, work with a certificate (even if self-signed) on the client end. Find the directory containing file ca-certificates.crt. You can find it with the locate command if necessary. On my computer that directory is /etc/ssl/certs. Then modify your openssl command to include the -CApath argument, as follows:

openssl s_client -connect imap.gmail.com:993 -quiet -CApath /etc/ssl/certs

Later in this document you'll see a this section on certificates: A Brief and Shallow Overview of Certificates

Because your response was in plain English when queried from openssl, this proves there's some ssl enabled service at port 993 of imap.gmail.com. Further, you can suspect it's IMAP, based on the reference to "Gimap" in the final line of its response to your connect request. As a matter of fact, if you'd done the same command with port 995, it would have referenced "Gpop" instead of "Gimap". On most servers, the OK line after the connection request gives you some hint of its service.

If you'd done the same command for port 143, it would have hung, because imap.gmail.com doesn't respond to anything, whether using using either openssl or telnet. Both commands hang. Now let's see if you try to access a non-ssl resource with openssl, but let's start with telnet just to prove there's something there:

slitt@mydesk:~$ telnet imap.earthlink.net 143
Trying 209.86.93.179...
Connected to imap.earthlink.net.
Escape character is '^]'.
* OK earthlink.net IMAP Service 19139 imapd EL_0_1_39_P at mp-venomous.atl.sa.earthlink.net ready

So the preceding proves there's something at imap.earthlink.net:143, it's readable in plain text telnet, making it likely that it's not ssl (although maybe it might work with STARTTLS). Now let's try to access it with openssl:

slitt@mydesk:~$ openssl s_client -connect imap.earthlink.net:143 -quiet
139951306700480:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:766:
slitt@mydesk:~$ 

The preceding response is what you typically get if you try to openssl connect with a valid service that's either plain text or some format openssl doesn't understand. If you try the same command with a port that's firewalled off or has no program listening, you get either a hang or the following:

slitt@mydesk:~$ openssl s_client -connect 127.0.0.1:54321 -quiet
connect: Connection refused
connect:errno=111
slitt@mydesk:~$

If you suspect something is plain text, you can investigate further with telnet:

slitt@mydesk:~$ telnet imap.earthlink.net 143
Trying 209.86.93.175...
Connected to imap.earthlink.net.
Escape character is '^]'.
* OK earthlink.net IMAP Service 28246 imapd EL_0_1_39_P at mp-grizzled.atl.sa.earthlink.net ready

If you telnet to an ssl-enabled port, you'll get connected, but without any handshaking, nothing else, and especially no OK message. Consider the following:

slitt@mydesk:~$ telnet imap.gmail.com 993
Trying 173.194.64.108...
Connected to gmail-imap.l.google.com.
Escape character is '^]'.
slitt@mydesk:~$

If you compare the preceding to its openssl equivalent (and use the browser's back button to return here), the lack of an OK message is clear.

Some IMAP servers advertise their capabilities. Consider the following openssl session with my local Dovecot server:

slitt@mydesk:~$ openssl s_client -connect 127.0.0.1:993 -quiet
depth=0 O = Dovecot mail server, OU = mydesk, CN = mydesk, emailAddress = root@mydesk
verify error:num=18:self signed certificate
verify return:1
depth=0 O = Dovecot mail server, OU = mydesk, CN = mydesk, emailAddress = root@mydesk
verify return:1
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Steves Dovecot IMAP Server welcomes you, do your thing

Steve Litt is the author of Twenty Eight Tales of Troubleshooting. Steve can be reached at his email address.

Logging In

Once you get the connection OK message, it's time to log in.

slitt@mydesk:~$ openssl s_client -connect imap.gmail.com:993 -quiet
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
* OK Gimap ready for requests from 68.204.228.160 ba7if2687949oab.268
a1 login yourname yourpass
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH
a1 OK yourname@gmail.com Your Name authenticated (Success)

What happened in the preceding session is that first, you connected, proving the URL and port were correct, and that this is an SSL/TLS environment under which you can operate. So the remaining questions are, what are your username and password? Here's where you try both myname@myisp.com and myname, as well as any other username constructs suggested by the ISP's documentation or tech support personnel. The benefit here is you've already proven that you have the correct URL and port, and you're very likely dealing with an SSL/TLS connection, so username and password are the only variables you need to deal with. Even if you have to use trial and error, it will be a matter of minutes, not hours.

Just so you know what login failure looks like, here's a login attempt using a valid username but wrong password:

slitt@mydesk:~$ openssl s_client -connect imap.gmail.com:993 -quiet
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
* OK Gimap ready for requests from 68.204.228.160 ga20if5116961oab.32
a1 login littdom wrongpass
a1 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)

And here's what it looks like when you use a username that doesn't exist:

slitt@mydesk:~$ openssl s_client -connect imap.gmail.com:993 -quiet
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
* OK Gimap ready for requests from 68.204.228.160 x6if5068626oay.152
a1 login noexistuser mootpass
a1 NO Invalid credentials x6if5068626oay.152

As you can see, both return "Invalid credentials", but a correct username additionally delivers "[AUTHENTICATIONFAILED]". So finally, you have a way of telling whether you have a correct (or at least existing) username, and if you do, the problem is the password.

Steve Litt is available to select clients to personally teach the Universal Troubleshooting Process Course. Steve can be reached at his email address.

Looking at Your Account With openssl

The preceding section, Logging In, has almost certainly given you everything you need to set your email client to interact with your IMAP account. And most of the time, once your email client is logged into your IMAP account, you can check and troubleshoot IMAP interaction directly from your email client. But perhaps, in certain circumstances, you might prefer to troubleshoot from an openssl session. If that ever happens, this section tells you how.

Danger Will Robinson!

Be very careful when manipulating your IMAP mail account with openssl. Do the wrong thing and you could irretrievably lose messages, mailboxes, or the whole account. I'd highly advise you to stay away from the SELECT and DELETE commands, and be very careful with the RENAME command. Even then, it could still be dangerous. Use your email client for troubleshooting unless that doesn't work, or you don't trust your email client to keep your IMAP account data safe.

Read-only Session

The following is an IMAP session that reads info out of my Dovecot IMAP. The meanings and results of each command are discussed later in this section:

slitt@mydesk:~$ openssl s_client -connect 127.0.0.1:993 -quiet
depth=0 O = Dovecot mail server, OU = mydesk, CN = mydesk, emailAddress = root@mydesk
verify error:num=18:self signed certificate
verify return:1
depth=0 O = Dovecot mail server, OU = mydesk, CN = mydesk, emailAddress = root@mydesk
verify return:1
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Steves Dovecot IMAP Server welcomes you, do your thing
aa login slitt slitt_password
aa OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAYLine continues
THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1Line continues
CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE] Logged in
ab list "" "%test%"
* LIST (\HasNoChildren) "/" "litttest_gmail"
* LIST (\HasNoChildren) "/" "testt"
* LIST (\HasChildren) "/" "litttest"
* LIST (\HasNoChildren) "/" "test2"
ab OK List completed.
ac examine testt
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS ()] Read-only mailbox.
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1327304935] UIDs valid
* OK [UIDNEXT 5] Predicted next UID
* OK [NOMODSEQ] No permanent modsequences
ac OK [READ-ONLY] Select completed.
ad search return (save) since 1-Jan-2014
ad OK Search completed (0.000 secs).
ae fetch $ body.peek[header.fields (from date subject)]
* 2 FETCH (BODY[HEADER.FIELDS (FROM DATE SUBJECT)] {166}
Subject: [ENROLL] My NO Hype, NO BS IM Training...Everything I Know!
Date: Thu, 16 Jan 2014 17:58:13 -0500
From: Mark Thompson <support@sixfiguremastermind.com>

)
* 3 FETCH (BODY[HEADER.FIELDS (FROM DATE SUBJECT)] {196}
Date: Thu, 16 Jan 2014 18:10:38 -0500
From: "Steve Litt of Troubleshooters.Com" <littdom@gmail.com>
Subject: test message, 20140116_1809 from littdom@gmail.com to
 slitt@troubleshooters.com

)
* 4 FETCH (BODY[HEADER.FIELDS (FROM DATE SUBJECT)] {113}
Subject: Uncaught bounce notification
From: mailman-bounces@golug.org
Date: Thu, 16 Jan 2014 18:52:51 -0500

)
ae OK Fetch completed.
af fetch 3 body.peek[header.fields (from date subject)]
* 3 FETCH (BODY[HEADER.FIELDS (FROM DATE SUBJECT)] {196}
Date: Thu, 16 Jan 2014 18:10:38 -0500
From: "Steve Litt of Troubleshooters.Com" <littdom@gmail.com>
Subject: test message, 20140116_1809 from littdom@gmail.com to
 slitt@troubleshooters.com

)
af OK Fetch completed.
ag fetch 3 body.peek[text]
* 3 FETCH (BODY[TEXT] {116}
This is a text message, timestamp 20140116_1809, emailed from
littdom@gmail.com to slitt@troubleshooters.com.


)
ag OK Fetch completed.
ah logout
* BYE Logging out
ah OK Logout completed.
slitt@mydesk:~$ 

This was a long session, but I'll break it down. Please keep reading:

aa login slitt slitt_password

This is how you log into an IMAP server. IDstring followed by the login command followed by the username and password. If things went well, you get an OK message. My Dovecot server also gives a list of capabilities, starting with IMAP4rev1, which is a pretty clear statement that this is an IMAP session.

ab list "" "%test%"

Here you're searching for any mailboxes in the root level that contain the word "test". Unfortunately, the list command is case sensitive, and I've found no way to make it case insensitive, meaning that, to search for "Test" or "test" you'd need to either do multiple commands, or put the whole list through a case-insensitive sort and select of your making. But unfortunately, this whole session takes place inside the openssl program, so there's no easy way to reach in and do your own processing.

Some important wildcards in search are:

Different combinations of these two wildcards together with text can produce some pretty specific results, but, as mentioned before, everything's case sensitive. I'm not going to extensively document these wildcards. Experiment and see for yourself.

ac examine testt

After having listed everything beginning with "test" on the top level, we've decided to look at folder testt. The examine command is read-only, it doesn't allow adding or deleting messages. This command returns a lot of good info, include the number of messages contained, and the number "recent" messages, whatever that means. The number of messages gives you a good idea of the range of valid message numbers for the fetch command. Further searches and fetches operate on this folder.

ad search return (save) since 1-Jan-2014

The search command searches on a criteria, in this case "since 1-Jan-2014". You need to put dates in Unix format. The return(save) part of the command saves the returned results to a variable called $ for later use. Please remember this, because it's not well documented elsewhere on the web.

Note

Don't confuse the SEARCH command with the LIST command. SEARCH returns messages within a folder, but LIST returns a list of folders.

ae fetch $ body.peek[header.fields (from date subject)]

Oh man, where do I start? Complicated command. In this command, instead of fetching a specific message number, $ fetches every message returned and saved from the preceding search.

The word "body" is misleading

The word "body" is misleading, because it can return any header item, any non-header item, or the thing most people call the "body". To return the thing most people call the "body", do this:

idstring fetch msgnumber body.peek[text]

Where idstring is an something like aa or ab or whatever, and msgnumber is the message number of the message you want to look at.

Meanwhile, body.peek is a special form of the body command that does not mark the fetched message as having been read. You don't want your tests to change anything.

[header.fields (from date subject)] part means to return and display header items "from", "date", and "subject". This is a nice, quick way to figure out what message you want to look at.

af fetch 3 body.peek[header.fields (from date subject)]

Here, instead of fetching everything returned from the search, you return from, date, and subject only from the message you found interesting and picked, in this case message 3. The only reason you did this is to print relevant header info before you print the body.

ag fetch 3 body.peek[text]

This returns and displays the text of the body, and only the text of the body.

ah logout

This logs you out of the IMAP session, and once that happens, openssl terminates, leaving you at the command prompt.

Lessons Learned

In this section, you've pretty much learned to navigate your way around your account, in a read-only fashion, using openssl and several IMAP commands. Remember to use EXAMINE instead of select so that you read and don't change anything. For the same reason, use BODY.PEEK instead of just BODY.

Steve Litt has written a large collection of Lyx documentation. Steve can be reached at his email address.

Modifying Your IMAP

Sometimes you want to write to your IMAP account and test whether your mail client sees it. This is especially true when you're wondering whether you're looking at IMAP or cache with your email client. So, to modify your account, while logged into IMAP through openssl, do the following, assuming you don't currently have a junk5 folder in the top level:

ab create junk5/junk6/junk7
ab OK Create completed.
ac list "" "%junk5*"
* LIST (\HasNoChildren) "/" "junk5/junk6/junk7"
ac OK List completed.

The preceding two commands first created a folder tree with junk7 under junk6 under junk5, and then verified that the whole tree had been created. The IMAP CREATE command can make an entire tree at once, without requiring the parent folders to already be created.

Now go to your email client. Most email clients don't faithfully conform to the entire IMAP specification, so it's very likely you'll need to refresh your client's directory tree for the account. In Claws-Mail, you right click the account in the tree, and then choose "Check for new folders". If everything's right, you should see top level folder junk5 containing junk6 containing junk7. That proves you made the change directly in IMAP.

Now, to limit the risk in deleting something, instead of doing a deletion within your openssl IMAP session, delete the directory tree with your email client. The exact method for such deletion varies from client to client.

Warning

Your email client might display wrong information upon such delete. For instance, Claws-Mail says it can't delete junk5. However, after the supposedly unsuccessful attempt to delete junk5, the following check within the openssl IMAP session indicates the deletion was indeed successful:

ad list "" "*junk5*"
ad OK List completed.

And, in fact, doing another "Check for new folders" in Claws-Mail reveals that junk5 and the tree underneath it are indeed gone.

This is one reason why directly manipulating your IMAP without an email client is so valuable. It's more likely to give you accurate information.

Confidence comes from modifying your IMAP account in openssl and having the results show up in your email client. Better yet, if you ever have a situation where you can log in with your email client but things go wrong, whether with reading or writing, now you know how to directly read or directly inject things into your IMAP account.

Steve Litt is the author of Troubleshooting: Just the Facts. Steve can be reached at his email address.

A Brief and Shallow Overview of Certificates

Certificates are a huge topic. A topic for which I have so little knowledge that I can barely scratch the surface. So if you're paranoid or even seriously interested in security, this section doesn't cut it: You'll need to learn more about certificates from other sources, probably by searching the web.

The idea behind a certificate is that it's issued by a Certificate Authority (CA), that vouches for the identity of the person or server holding the certificate. In other words, if, during an online interaction, I display a certificate given me by Verisign, it says that I did enough to identify myself or my server to Verisign that they now believe strongly enough that I'm Steve Litt, or Troubleshooters.Com, or whatever, that they're willing to issue me a certificate saying I'm who I say I am (or my domain or whatever). The idea is, other entities will see my certificate and say "if Verisign says he's Steve Litt, then I believe it." Imperfect, but better than trusting me on my say-so.

The idea of trusting me on my say-so brings up the next topic: Self-signed certificates. A self-signed certificate has a similar structure to one issued by a CA, but I created it. I'm saying I'm Steve Litt: Truuuust me! A CA issued certificate is like a state-issued drivers license, whereas a self-signed certificate is more like a piece of ID I created myself. Or perhaps a business card. It has all my information, always assuming I'm who I say I am.

If a CA issued certificate is like a drivers license, and a self-signed certificate is like a business card, having no certificate at all is like having no identification at all. It's sketch. It's unprofessional.

Why Is Self-Signed Better Than None At All?

Good question. Neither proves your identity. But a self-signed certificate at least enables you to interact, without error, with Internet resources that demand some sort of certificate.

No Certificate

Here's what happens when you sign into Gmail with no certificate:

slitt@mydesk:~$ openssl s_client -connect imap.gmail.com:993 -quiet
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
* OK Gimap ready for requests from 68.204.228.160 ek7if15334870oab.213

Self Signed Certificate

Here's what happens when you sign into Gmail with a self-signed certificate:

slitt@mydesk:~$ openssl s_client -connect imap.gmail.com:993 -quiet -CApath /etc/ssl/certs
depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = imap.gmail.com
verify return:1
* OK Gimap ready for requests from 68.204.228.160 w8if15379320oax.104

Here's the Point

As you can see from the preceding, by declaring the self-signed certificate in /etc/ssl/certs/, I was able to connect with imap.gmail.com without an auth error, and I was able to see its certificates. Not every IMAP server works like this. My local Dovecot IMAP server errors out when the client presents a self-signed certificate:

slitt@mydesk:~$ openssl s_client -connect 127.0.0.1:993 -quiet -CApath /etc/ssl/certs
depth=0 O = Dovecot mail server, OU = mydesk, CN = mydesk, emailAddress = root@mydesk
verify error:num=18:self signed certificate
verify return:1
depth=0 O = Dovecot mail server, OU = mydesk, CN = mydesk, emailAddress = root@mydesk
verify return:1
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Steves Dovecot IMAP Server welcomes you, do your thing

Getting your Dovecot IMAP to interact completely with clients' self-signed certificates, if that's what you want to do, is beyond the scope of this document. To figure out what to do, start at http://wiki2.dovecot.org/SSL/DovecotConfiguration, and then search the web.

There's another reason a self-signed certificate is better than none at all. With no certificate, every communication is insecure. But if there's a self-signed certificate, if the first communication is between the true and correct entities, and if both entities do their job and save their counterparts' certificates, then all future communications between them will be secure, until one or both change their certificates.

You Can Now Get Free Certificates

This is good news for security, although taking advantage of it requires some more knowledge (which you can get in other documents, via web search).

Start by reading this: https://konklone.com/post/switch-to-https-now-for-free?hn. The preceding URL refers to getting your certificate from startssl.com.

The information in this section on certificates is not authoritative, and very likely not even completely correct. This section's main purpose is to let you know that certificates play a big role in security, and that there's more information out there.

Steve Litt oversees content at Troubleshooters.Com. Steve can be reached at his email address.

When Your Dovecot IMAP Doesn't Work

So far this document has dealt with troubleshooting when the IMAP server is known to be good. Knowing how to troubleshoot with a known good IMAP server is necessary, because a malfunctioning Dovecot IMAP server absolutely needs to be troubleshot with IMAP commands, following the troubleshooting hierarchy described in Connecting Your Email Client to an IMAP Server:

The rest of this document describes troubleshooting tools and techniques used to fix an ailing Dovecot IMAP server under your control.

Steve Litt is the author of several human performance books. Steve can be reached at his email address.

Handy Dovecot Commands

The following commands are essential to Dovecot IMAP troubleshooting:

dovecot reload

Changing the Dovecot config file (/etc/dovecot/dovecot.conf by default), along with /etc/dovecot/local.conf and /etc/dovecot/conf.d/*), does not change the behavior of the running copy of Dovecot. No problem, because the following command adjusts the running copy to conform to the (newly updated) configuration:

dovecot reload

So, to avoid the kind of troubleshooting mistakes that turn minutes into hours, always make it a habit to perform the preceding command every time you change Dovecot's configuration.

dovecot -a

This command prints out every single configuration option of your running copy of Dovecot (remember to reload), whether the config option was set explicitly in the config files, or acquired via Dovecot program defaults. Config file comments don't appear in this output, and this output reflects the actual effect of all your config files. This is a heck of a lot of information, often overkill, but it's good for getting ideas when your troubleshooting seems to be at a dead end.

dovecot -n

This command prints only those config options that are different from the defaults of the dovecot program. This is of huge value because it's usually a very small output, and typically shows the things you changed. This command makes your changes instantly highly visible. This is the best command for quickly seeing what Dovecot is really doing. If a config setting in this output is different from what you think you set it to within the config files, you can troubleshoot that.

dovecot stop

You might want to do this if for some reason you want to stop Dovecot entirely, or want to stop it and then run it, in the foreground, with an alternate config file

Here's the result of a non-running Dovecot on my machine:

slitt@mydesk:~$ openssl s_client -connect 127.0.0.1:993 -quiet
connect: Connection refused
connect:errno=111
slitt@mydesk:~$ 

Well of course the connection's refused. Without Dovecot running, there's nothing listening on port 993.

Warning

On many systems, Dovecot is run by one of those systems that restarts it when it stops. You can tell with a ps ax | grep dovecot command, observing the process IDs. If that's the case, and you truly want to stop Dovecot, you'll need to use that system's stop command. For instance, if it was started by Ubuntu's upstart system's stop command to stop it, and its start command to start it again. Likewise, if Dovecot were started by DJB's daemontools system, you'd stop it with the svc -d command, and start it up again with its svc -u command. If there are any of those SYSV systems still around, you'd use something like /etc/init.d/dovecot stop and /etc/init.d/dovecot stop.

dovecot -c config_file

This starts Dovecot with the config file of your choice, and is much more useful than you might think.

Steve Litt is the author of Rapid Learning for the 21st Century. Steve can be reached at his email address.

IMAP Log Files

I see two problems with Linux log files:

  1. Finding the right log file
  2. Interpreting the log entries

What I do is when I get an error or unexpected behavior in an IMAP or other email activity, I assume that the relevant log file is in /var/log, so I get in that directory and perform the following command:

ls -ltr

Here are the last (latest) four entries I got for that command:

-rw-r----- 1 syslog            adm        12610 Jan 17 19:57 mail.err
-rw-r----- 1 syslog            adm      2247455 Jan 17 20:00 mail.log
-rw-r----- 1 syslog            adm        27336 Jan 17 20:17 syslog
-rw-r----- 1 syslog            adm       369210 Jan 17 20:20 auth.log

Of those four files, I've found mail.err the most handy in solving IMAP problems, but your mileage may vary. The following is the mail.err entry that helped me solve a reproducible IMAP problem that had tortured me for hours:

Jan 14 14:36:07 mydesk dovecot: imap(slitt): Error: user slitt: Initialization failed: namespace configuration error: Duplicate namespace prefix: ""

For hours I'd assumed my problem was with ports, authentication, SSL/TLS settings, and the like. But after seeing this error message, and then running a tail -f on mail.err and watch that exact log entry written every time I failed to connect, I understood that Dovecot didn't like the way I was doing namespaces, and all the work I'd been doing to deduce my username/password/auth/port/SSL/TLS had been futile. I was looking in the wrong place. For any of you who have taken my course or read my books, you know I had made The Big Mistake. But at least within a few hours I'd found and corrected my bad assumption.

Notice that the mail.err line shown earlier had a timestamp. Always use those timestamps to deduce whether the error pertained to something you did, but noting the time when you fail at something, and look around that time in the log. And of course, you can see log entries being written in real time by opening a separate terminal window and performing the following command (probably as root):

tail -fn0 mail.err

Arrange your workspace so the window tailing the log file is on one side, the window in which you're performing a (failing) IMAP activity is on the other side, perform the command, and note the log file entry written.

When dealing with log file entries, read the entry completely, thoroughly, and thoughtfully. Often it tells you exactly where the error is, if you read it carefully enough and think enough about it.

Steve Litt is the originator of the Universal Troubleshooting Process. Steve can be reached at his email address.

Dovecot IMAP Troubleshooting Techniques and Mindset

What could possibly go wrong? You have IMAP running on top of SSL/TLS, maybe or maybe not with STARTTLS, all of which runs over TCP/IP. You have an email client that probably looks like a black box to you, and a Dovecot server whose config files are so insanely complex and numerous that your Dovecot server seems like a black box too. Meanwhile, misinformation abounds, with some saying port 993 guarantees an SSL connection (they never call it TLS, only your email client's setup calls it that), while others say port assignments are just a matter of custom. TLS is sometimes, but not always, referred to as SSL, and STARTTLS is some new fangled thing that's either better or worse than SSL. You look on the Internet, but for every piece of reasonable information, you find ten pieces of either incomplete assumptions or half-baked fakery. When it comes time to troubleshoot your Dovecot IMAP server, in the words of Thomas Paine, "These are the times that try men's souls."

Thomas Paine continues: "The summer soldier and the sunshine patriot will, in this crisis, shrink from the service of their country". Likewise, the good-times technologist, even the one so adept at spouting buzzwords and attaining certifications, when confronting IMAP problems, will quit, or try to have someone else fix it for him, or panic and turn a reproducible problem into a week long troubleshoot.

The best soldier isn't the strongest man or the best marksman, he or she is the one with the right mindset. Plenty of crack shots deserted when the British invaded.

Same with troubleshooting IMAP (or anything else). Technical knowledge alone doesn't cut it, plenty of technical geniuses fail miserably at troubleshooting, and technical knowledge isn't as important as mindset. I'd go so far as to say that the technical information on this web page is adequate to solve most Dovecot IMAP problems, if you have the mindset. If you're not familiar with what I teach about troubleshooting mindset, see these:

Techniques

We've covered mindset, now let's talk about techniques to help you implement that mindset.

Back Up

The moment you've installed Dovecot, before making any changes or even trying to run it, back up the /etc/dovecot tree, and save it forever. The two advantages you gain is that you can always "put it back" to the original configuration, and you can exploit the differences (see my Troubleshooting info on the web and books if you're not familiar with the power of exploiting the differences) between the current and original (or perhaps intermediate) settings.

If you've already modified the config

Perhaps you read this too late, and you already modified the config without first making a backup. As the old African proverb goes, the best time to plant a tree is twenty years ago. The next best time is right now. Go ahead and back up what you have right now. Then, if it's convenient and you think you can do it, remove all your changes, and make a second backup of that.

Touch nothing until you've made a good and restorable backup of /etc/dovecot.

Mark All Your Changes

A pro move is to mark all your changes, whether they're adds, changes or deletes. For instance, if the original config file expressly set first_valid_uid to 500, and this setting was not commented out, and you want to change it to 1000:

# first_valid_uid = 500           # slitt commented this out
first_valid_uid = 1000            # slitt added this

The preceding makes it clear that the former config file expressly configured first_valid_uid to 500, and I changed it to 1000. Another way, although without the audit trail value, would be the following:

first_valid_uid = 1000            # slitt changed this

If the original setting had been commented out, you wouldn't put your name on the line that was already commented out:

# first_valid_uid = 500 
first_valid_uid = 1000            # slitt added this

openssl as a Testpoint

As described in the first half of this document, openssl (or telnet if you're serving insecure IMAP) is a great way to look inside your IMAP server, and a great way to input a change. Once you find out it doesn't work with your email client, I suggest going right to openssl to save time, trial and error, and frustration.

Log Files

Another spectacular way to look inside the workings of your Dovecot IMAP server is to look at its logs as you exercise it with openssl. Read the log's error messages very carefully: They often point you straight at the problem.

Use dovecot reload then dovecot -n

Remember that any time you change anything in your Dovecot system's config files, you need to perform the following command, as root, to have them "take effect":

dovecot reload

Then run the following command to see how your Dovecot setup now varies from the default:

dovecot -n

Pro Move

The pros keep a log of config changes and the results of the newest dovecote -n. This makes it much easier to see what you've ruled out and what you haven't, and it makes it much easier to see the exact results of config changes.

Make Your Changes to local.conf

If you notice, the very bottom of a standard dovecot.conf contains the following line:

!include_try local.conf

What that line says is: "now load local.conf if it exists, and have anything in it overwrite anything that came before. This is a way you can leave the original config files alone, and just work on this one new file, making it less likely you'll confuse yourself. So, if you want to override the existing top level config item called first_valid_id, you'd simply do it in dovecot.conf

first_valid_uid = 1000

But what happens if you want to modify one item in a group? Perhaps you want to change the plain text imap port to 14300 (yes, this is a contrived example, and don't really do it, plain text is dangerous). Just do the following in local.conf:

service imap-login {
  inet_listener imap {
    port = 14300            # changed by slitt, don't really do this
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
}

So, to summarize, if you're making all your changes in local.conf, if modifying a lower level item, you need to reproduce the whole structure all the way up to the top level.

When You're Really Stumped, Use dovecot -a

If you're completely stumped, and have no idea how to narrow the root cause scope any further, it often helps to view the entire configuration as one file, without comments, instead of dovecot.conf, local.conf, and the 22 files inside conf.d, all of which seem to have more comments than config. To accomplish this, do the following:

dovecot -a | less

Go over each line, looking for ideas. It will take 15 to 30 minutes. But that's downright quick compared to blind trial and error or proceeding on an untested assumption.

The Nuclear Option using dovecot -n

Occasionally you might find a situation where the config Dovecot actually sees is not the config you have (or think you have) put in the configuration files. Or sometimes, the config perceived by Dovecot seems correct, but your log files show errors seemingly inconsistent with the configuration. You've been troubleshooting it for an hour, and now you're questioning your sanity. It might be time for the Nuclear Option:

mv dovecot.conf dovecot.conf.bup
dovecot -n > dovecot.conf

The preceding two commands backed up your original dovecot.conf and then replace it, and all the files it includes, by one very short dovecot.conf. It's a short file because it includes only those config options that differ from the program default. Now, instead of looking for a needle in a haystack, you're looking for the needle in a small ashtray containing 31 matches and a needle. Troubleshooting becomes much easier. How much? Consider the output of the following three commands:

slitt@mydesk:/etc/dovecot/conf.d$ dovecot -n | wc -l
32
slitt@mydesk:/etc/dovecot/conf.d$ wc -l ../dovecot.conf
58 ../dovecot.conf
slitt@mydesk:/etc/dovecot/conf.d$ cat *.conf | wc -l
1070
slitt@mydesk:/etc/dovecot/conf.d$

You've replaced 58 lines of dovecot.conf and 1070 lines of conf.d files with a new, 32 line dovecot.conf, all in a single file.

Once you fix the problem using your Nuclear Option dovecot.conf, you might wish to go back and put the fix into the original config tree, and restore the dovecot.conf that you backed up. Personally, I just kept the Nuclear Option dovecot.conf.

But doesn't that risk breakage if program defaults change?

When I kept the nuclear option dovecot.conf, I left my configuration dependent on the program defaults. If a later program version had different defaults, it could change some config that's vital to me. One could argue that there are some config options so important that one should strongarm them to their current default, to prevent later upgrades from setting them wrong. So here's what I did:

root@mydesk:/etc/dovecot# dovecot -a > dovecot_a.conf

Then I tested dovecot_a.conf like this:

root@mydesk:/etc/dovecot# stop dovecot
dovecot stop/waiting
root@mydesk:/etc/dovecot# dovecot -F -c frobo.gump
Fatal: Error reading configuration: stat(frobo.gump) failed: No such file or directory
root@mydesk:/etc/dovecot# dovecot -F -c dovecot_a.conf

Then I tested the Dovecot system with openssl, and when satisfied, I stopped the foreground Dovecot and did this:

root@mydesk:/etc/dovecot# start dovecot
dovecot start/running, process 10440
root@mydesk:/etc/dovecot#

So now, if I had wanted, I could have replaced the little dovecot.conf with a copy of dovecot_a.conf to guard against all default changes. I didn't do that, but I could have. What I am doing is to keep dovecot_a.conf around, so when I upgrade, or if my Dovecot breaks inexplicably in the future, I can run it with the dovecot_a.conf config file to see if that makes a difference, and how so. And in that future, I can run dovecot -n against Dovecot running the dovecot_a.conf config file, and diff that output to the little config file, to see what program defaults have changed.

Some Final Thoughts

I've barely scratched the surface of all the tests you can do on a Dovecot system. This document has revealed only a small fraction of the technology behind Dovecot. So when you troubleshoot Dovecot (or anything else), I want you to have the following belief firmly implanted in your mind:

My job isn't to fix it, my job is to narrow it down.

I've observed thousands of Troubleshooters. The best prediction of a Troubleshooter's productivity is whether he or she believes that "My job isn't to fix it, my job is to narrow it down." Those who strongly believe this, troubleshoot accurately and prolifically. Those who don't, are middling at best, mediocre or pitiful at worst.

And, in order to keep that retain this belief when things start falling apart, Ninja Troubleshooters ask themselves the following question every time they're stumped, frustrated or panicked:

How can I narrow it down just one more time?

The answer to the preceding question will lead you to the right diagnostic test, even if that test wasn't articulated in this document, and you don't know of that diagnostic test yet. After all, you can ask for help.

Steve Litt is the originator of the Universal Troubleshooting Process. Steve can be reached at his email address.

Asking For Help

Before asking for help from actual people, be sure you do at least a few minutes of web searching. Everyone has an online tech karma, and asking a question that could have easily been searched lowers your karma. Below a certain karma, you'll encounter hostility. Let your karma go lower, and you'll be asking questions into into an unanswering void. So do a few minutes research before asking real people.

If you haven't already, familiarize yourself with Eric Raymond's and Rick Moen's How To Ask Questions The Smart Way. A lot of this document is obvious, but a lot isn't. People asking questions in contradiction to this document are asking to be ignored, mocked or flamed. The rest of my Asking For Help section adds, emphasizes or contradicts what's written in "How To Ask Questions The Smart Way", and for the sake of brevity, for the rest of this section I'll refer to that document's contents simply as "Raymond".

Answers Are a Good Deed, Not a Duty

There's one in every mailing list and IRC channel: The guy who thinks he's owed an answer. He makes absolutely no effort to make his question easy to answer, and makes little effort at civility. Then he whines when nobody answers him, and if someone flames him for wasting everyone's time, he starts yelling and screaming, often saying we're not representing "our product" right, so it will fail.

Product? It's free software, poser, not a product. You want a product, pay me $500, sign the ten page EULA including indemnification and damages if you copy the program, and set yourself up for visits by the software police.

In response to the poser, some respondents flame the daylights out the poser, who is wasting brainpower and bandwidth. I applaud those who do. I usually just procmail them to /dev/null and don't worry about it again. Personally, I don't think anyone should answer these types, for the same reason I don't think a parent should buy the toy for the kid pitching a fit to try to get the toy.

Anyway, if you want a fun time asking questions, always remember that the person doing the answering is doing you a favor, so make it easy for him to answer, and be gracious. If you get no answers, try another mailing list, forum or IRC channel.

Examples Should Be Minimum Working Examples

Examples are sometimes necessary for other people to see what you're doing, or to reproduce your symptom. But there's a right and a wrong way to do it. The right way is a Minimum Working Example. Let me tell a story of a guy who refused to use Minimum Working Examples...

Before I email-filtered him out of my life, there was a guy on one of my mailing lists who specialized in asking dumb questions. One day he attached a document, submitted in an arcane and almost unused compression system, containing an entire directory, with three graphic images and a document with chapters, sections, and body text, as well as the images. His question? How do I center my sections?

Silly me, I actually spent 1/2 hour finding software that would read his arcane archive format, only to find a mess that would need to be reduced in order to efficiently examine the situation. He could have simply submitted a simple document with one section and nothing else, but that wasn't his style. On-list, I read him the rules of the road, in a rather neutral tone. Another guy asked him to submit a minimum working example. So he submitted another arcane format, this one ten times bigger than the first one.

The sad thing is, if he'd taken five minutes to make a copy of his document, gotten rid of everything except a section, the styles that give the section its appearance, and maybe a sentence of text, he could have sent it without archiving it, and gotten the answer within an hour. If, that is, the answer didn't become obvious to him once all the extraneous material was removed.

When submitting anything to others for help, always make sure to reduce what you submit to the smallest example that still reproduces the symptom. As a side benefit, reducing it to a minimal working example often makes the root cause obvious, enabling you to solve it without waiting for others.

IRC Channels Are Your Friend

IRC channels are those things you see with Xchat, Irssi, Quassel, or Kvirc. Most free software IRC channels are on the Freenode network, including #dovecot, #claws, and #mutt.

IRC Channel Rules of the Road

Always read the topic when you log in. The topic gives you the community's behavior expectations. For instance, #python frowns on people writing LOL, and #dovecot tells you to use ##imap for generic IMAP questions. If you need to read the topic again, the /topic IRC command will print it. Many topics refer you to a web page. Read it: You'll have a much better time on the channel if you know the rules of the road. Oh, and every topic forbids pasting output or files directly into IRC: Use a pastebin app, such as pastebinit or wgetpaste, to put it into a pastebin on the Internet, and then post the pastebin's URL to the IRC channel. IRC inhabitants get very testy when people paste directly to IRC, and I've seen people kicked off a channel for it.

Obviously, all the rules for email apply to IRC too: Be nice, consider an answer a favor, and follow the guidelines of Raymond's and Moen's How To Ask Questions The Smart Way.

Giving Help

Don't be that guy.

Every list, forum or channel has one. The neighborhood watchman. He watches posts like a hawk, replying nasty to every post that doesn't conform to his exacting standards. Starting with words of disapproval, he quickly transitions to fourth grade insults about IQ and mental stability if his expertise and authority aren't worshipped.

He's memorized every character of Raymond and Moen's How To Ask Questions The Smart Way except the How To Answer Questions in a Helpful Way, which says to do the opposite of everything he's doing.

Here are some of the misfortunes that befall that guy:

Because of two or three of these guys on the user mailing list of a well known Linux distribution, I stopped recommending that distribution. Even though the distribution is outstanding, I don't want those whom I refer subjected to the kind of abuse that passes for knowledge on that mailing list. Now I recommend a distribution that's not quite as stable or problem free, but whose mailing list treats questioners with respect.

Bottom line: Read and follow Raymond's How To Answer Questions in a Helpful Way, and if a poster gets under your skin, ignore him, or procmail him to /dev/null.

What About Repeat Offenders

The neighborhood watchman does himself and everyone else a disservice, but at some point, a repeated troll, a repeated proudly stupid, a repeated impatient insister, a crazyman, or a chronic offtopic chatterer must be read the rules of the road. Reading that person the riot doesn't make you a neighborhood watchman, it makes you someone looking out for tech communications in your chosen information venue.

That being said, reading the rules of the road is something best done once. If the guy doesn't take the advice, just procmail him to /dev/null. Your life will be much happier: Some people just can't be taught.

Steve Litt is the author of Thought Patterns of the Troubleshooting (and Debugging) Ninja. Steve can be reached at his email address.

Things Not Covered

Obviously, this document just scratched the surface. Multiple namespaces? Not described. Public namespaces? Not described. Authentication methods other than PAM? Not described. Ramifications of using . as a separator instead of /? Not described. This document's coverage of certificates was cursory and probably not entirely accurate. I'm an IMAP newbie, so I'll bet you dollars to donuts there are twenty IMAP topics I didn't cover because I didn't know they exist. Plus, of course, Dovecot can be used for POP3, and I didn't cover that at all.

Nevertheless, using the info on this website, the Universal Troubleshooting Process, and the mindset of troubleshooting, plus a little web research and maybe asking for help, I'll bet you dollars to donuts you can solve any reproducible IMAP problem quickly.

Steve Litt has documented several aspects of transportational bicycle riding. Steve can be reached at his email address.

Personal Dovecot

This section is a description of the benefits of a personal, local, Dovecot IMAP server rather than how to troubleshoot, but I believe this document would be incomplete without this section.

If you've ever been forced to change email clients, you know what a catastrophe moving your maiboxes can be. That's why nowadays I never store my emails on my email client. Instead, I store them on my personal Dovecot server on my desktop machine. Now I can switch email clients every five minutes. I can switch them as a troubleshooting diagnostic test. No email client will ever again own my data.

As an added benefit, my email filtering is done via Procmail, and my .procmailrc is in my backed up home directory, not somewhere in /etc where it gets overwritten every time I switch distros or even versions (I reinstall, I don't upgrade).

The following is an approximate diagram of my setup:

Diagram of my Dovecot setup

If you want to learn more about setting up a personal Dovecot, see my Escape from Kmail document.

Steve Litt is the originator of the Universal Troubleshooting Process. Steve can be reached at his email address.