The Symptom

If you use Windows 95, 98 or ME, it will happen to you sooner or later -- Win9x access to Samba servers will fail. It will typically happen right after an upgrade to the Samba server. The symptom will probably look like the typical "need password for IPC$" dialog box:
Screenshot of the symptom

If people have already been messing around trying to fix it after the upgrade, you might also get the "Not Accessible" dialog box as shown below:
The Not Accessible dialog box

I know you've seen symptoms like these before, but typical as they look, you'll find this symptom terribly difficult to troubleshoot because of the huge number of variables and because the solution requires not only four actions, but those four actions must be taken in a certain order. This document has been written to save you hours of troubleshooting, experimentation and just plain trial and error.

The root cause is simple enough. Samba version 3.2.0 turned off lanman support by default, both on the client (client lanman auth=) and the server (lanman auth=). 3.2.0 also turned off the default for client plaintext auth=. Before 3.2.0 all three were turned on by default. Windows 9x can only work with lanman support, so unless a corresponding modification of smb.conf is done before the first login, there will be Win9x access failures.

The Solution

There's a proactive solution and an after the fact solution:

Proactive Solution

The proactive solution is trivial. While still running a Samba older than 3.2.0, simply put the following three lines in the [global] section of your smb.conf:

lanman auth = Yes
client lanman auth = Yes
client plaintext auth = Yes

These will be redundant while still using pre 3.2.0 because they re-state the defaults, but the second you switch to 3.2.0 or higher, these keep your lanman auth working, so that your Windows 9x access remains unchanged.

NOTE
There have been some reports that the following line might also be needed:

client ntlmv2 auth = no

I haven't reproduced this, but I'd suggest you add this as a fourth line for the following reasons:

Win9x cannot authenticate via NTLM, according to the documentation
client ntlmv2 auth defaults to "no", according to the documentation

The After the Fact Solution

Unfortunately this problem isn't universally known, so it's very possible your first introduction will be after the problem occurs. In that case, you can fix it by following the following process exactly:

Add these three lines to the [globals] section of your smb.conf:

lanman auth = Yes
client lanman auth = Yes
client plaintext auth = Yes

Restart the Samba server
Re input passwords for every Win9x user:
smbpasswd -a username
Have every Win9x user log out and then log in again

Take it from me: You need to do all four steps in order. Every time a user logs into a Samba server with these three properties set to "no", whether by default or explicitly, the lanman part of the user's password is erased and replaced with X's, meaning even after the properties are enabled, that user still can't log in via lanman, and for Win9x and before, lanman is the only game in town.

Is Rebooting Necessary?

Several people have pointed out that step #2, rebooting, is not strictly necessary. All that's needed is for your server to recognize the changes you made to smb.conf so that step 3, re-inputting the passwords, will write Lanman passwords. In fact you can send a SIGHUP signal to the smbd daemon, or at least that's the way it used to work:

kill -SIGHUP <smbd pid>

Trouble is, modern Samba servers have multiple smbd daemons running. Which one do you SIGHUP? In my opinion, unless your Samba network is being used enough that a few "log off Samba" calls won't do the trick, why not restart Samba. That way you KNOW it reread the smb.conf file. Also, if there's significant troubleshooting to be done, you want people off the Samba network so they don't keep losing their work.

So, in my opinion, except in a full production environment, it's better that step 2 be an actual Samba restart rather than a SIGHUP. Your mileage may vary.

Thank You to the Samba Team!

This document isn't an idle academic treatise. It happened to me and took hours to troubleshoot because of the nature of the system, which will be explained in the next article, The Theory Behind. After several hours I admitted my inability to solve this myself in a reasonable timeframe and emailed the Samba list. 39 minutes later Samba's John Terpstra replied with the exact root cause, that Lanman had been defaulted to "no" in the new version and it was needed by Win9x clients. His email also identified the consequential damage needing undoing, the overwritten password.

39 tiny minutes. Just try getting that kind of support from a commercial vendor!

John and the whole Samba team, Thank You! Thanks for the quick and accurate help, and thanks for making a wonderful and useful piece of software.

The Theory Behind

The preceding sections titled The Symptom and The Solution give you a quick symptom description and a quick solution. If you wonder why the four steps must be done in order, read this section.

Lanman is a weak form of encryption. It was used for directory shares by Windows 95, Windows 98, and Windows ME. Later versions of Windows are able to use better forms of encryption, so clients of later versions should use those better versions, even though Lanman is supported for backward compatibility at least as late as Windows Vista.

If there are NO pre-XP Windows clients on your network, then this document does not apply to you. But if there's even one client that's Win95, Win98 or WinME, then if you want that client to have access to your Samba server, your Samba server must support Lanman.

Before Samba version 3.2.0, Samba supported Lanman by default. If you didn't want it, you had to turn it off explicitly. Somewhere around the summer of 2008 version 3.2.0 became available. However, it took several months for the new version to be incorporated by various Linux distributions, and then it took a few more months for those Linux distributions to be installed. The result is that for 2008 and the first couple months of 2009 you were unlikely to encounter Samba 3.2.0, so there were few reports of Windows 9x client Samba failures. Mid and late 2009 will see an increase as the few remaining Windows 98 users encounter Linux distros with Samba 3.2.0.

As mentioned, Windows 9x and ME have only one encryption choice: Lanman. If Lanman is turned off, they cannot get shares from the server. Samba 3.2.0 and newer turn Lanman off by default, so when a newer Samba is installed, Windows 98 clients lose their ability to get shares. A Windows client communicating with a non-Lanman Samba server repeatedly asks for the password for the IPC$ resource, and no amount of typing in the password will satify it. The symptom looks like this:

Screenshot of the symptom

You might think the solution is simply to enable Lanman using the following three smb.conf settings:

lanman auth = Yes
client lanman auth = Yes
client plaintext auth = Yes

That's only part of the solution. Obviously the second part of the solution is to restart Samba so those changes are incorporated in the running server. But even that is not enough. Here's why...

The password records on the Samba server, whether kept in the /etc/samba/smbpasswd file or in one of the newer Samba password databases, contains two password entries: one Lanman style entry, and one NT style entry (XP and later). During the time the Samba server was running without Lanman, every time a Windows client tried to log into a Samba share on that server, the Lanman entry was erased and replaced with X's. What that means is even after the Samba server is correct, some or all of its Lanman password entries are invalid because of damage done during the time Samba was running without Lanman support, so those Win9x and ME users STILL can't log into the PERFECTLY FUNCTIONAL Samba server.

Exhibit A: /etc/samba/smbpasswd entry before and after logging into non-Lanman Samba server the first time.
Before	james:1508:850CE963E2B8BF87451F0ABF29454BA6:278AC243ABDE22263F8C90EDD3793C82:[U ]:LCT-6A8B3725:
After	james:1508:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:278AC243ABDE22263F8C90EDD3793C82:[U ]:LCT-6A8C3B16:
NOTE: The preceding are just a simulation and do not reflect anyone's actual password data. Notice that the NT password, beginning with 278A, remains the same, but the Lanman password, which began with 850C, was replaced with X's.

Here's where things can go bad fast if you don't have prior knowledge of the symptom and solution. Not knowing this information, Lanman status is just one of a hundred possible root causes that can be imagined. If you're anything like me, when you see that changing the Lanman status doesn't work, you think you ruled it out, go on to other things, and probably you'll remove those three lines so as not to complicate your smb.conf. UGH!

You just lost an hour or more, because you'll diagnose other parts of the system before wondering whether a test or conclusion led you in the wrong direction. And it gets even worse...

The Troubleshooter can accidentally get the four steps in order and it works for awhile until he undoes the three smb.conf options. And when he does, it doesn't show up until the Windows 9x/ME users log out and log back in. And if a W95/ME user logs into a Lanman speaking Samba server that he also logged into before it spoke Lanman, his symptom would change:

From this

To this

In other words, as troubleshooting proceeded, this problem would start to look very intermittent.

Oh, and by the way, just to add to the mix, using smbpasswd to update or add a password while Lanman is disabled has the same effect as logging in at that time -- it X's out any previous Lanman password.

Ways This Could be Troubleshot from Scratch

It's ugly, but it could be done. I got pretty close before asking for and receiving help from the Samba mailing list.

You could quickly prove that a tiny smb.conf works perfectly with smbclient on Linux but fails miserably on Win98. You can grab data using smbclient from other Linux boxes, thereby ruling out the network and firewall. If you have several WinXP/Vista clients and several Win98 or ME clients, you could quickly show that it's a Win98 thing. Win95 is iffy, because some Win95 boxes still speak plain text over the wire rather than Lanman. The fact that ALL the Win98 clients exhibit this problem shows that although it's a Win98 problem, the solution will probably be on the Samba end, unless there's some esoteric registry hack to be done.

When I was working this out, I very quickly suspected new defaults on the new Samba version caused the malfunction. So I used the following to figure out any changes:

testparm -vs

The preceding prints out every possible option and its setting, whether it was set explicitly or whether it was simply a default. I had a Mandriva 2008.1 with Samba 3.0.28a computer that worked with a tiny smb.conf, and a Ubuntu 9.04 computer with Samba 3.3.2 that failed with the same tiny smb.conf with its netbios name changed to reflect the hostname. I ran the testparm -vs using the tiny smb.conf on both systems, piping them into files. Then I and ran the diff command on both files and came out with 93 lines of differentials. Excluding the obviously changed netbios names and comments, excluding options occurring in one file but not the other, and excluding extremely esoteric options that were less likely to apply, the following stuck out like a sore thumb:

43c42
<       lanman auth = No
---
>       lanman auth = Yes
46,47c45,46
<       client lanman auth = No
<       client plaintext auth = No
---
>       client lanman auth = Yes
>       client plaintext auth = Yes

So I inserted:

lanman auth = Yes
client lanman auth = Yes
client plaintext auth = Yes

and restarted Samba and tried again. I'd like to tell you that was the turning point but when it didn't work (because I didn't redo the passwords and on the Windows box didn't log out and back in again), I concluded that wasn't the cause, removed those lines from smb.conf to make it simpler, and went on looking elsewhere.

The diff command wasn't a total loss though -- when I finally emailed the Samba list and John H Terpstra let me know about the need to re-password, I knew exactly what he was talking about due to the diff command, and from there it took less than 10 minutes to not only fix the problem, but to repeatedly toggle the symptom on and off to make sure I wasn't dealing with an intermittent.

Summary

Logging into a non-Lanman server from a Win9x client overwrites the Lanman part of the password record with X characters. So does refreshing the password with smbclient when the server is set not to speak Lanman. This is consequential damage that must be undone after the root cause (lack of Lanman) is fixed, but unfortunately what it looks like is that you didn't fix the problem. Added to that is the fact that if the Win9x client has a connection, server password changes won't matter until the Win9x client logs out and in again. All this adds up to what looks like a nasty intermittent, but really simply requires the root cause to be fixed, the consequential damage (overwritten server side password) to be fixed, and a known state to be acquired (log out and in on the Win9x box).

Material Not Covered

This document should help most who encounter Win9x Samba problems after upgrading to or past Samba 3.2.0. It contains the general principles you need to fix this problem, and if you read this before upgrading to or past 3.2.0, how to avoid this problem completely.

But there are some things not discussed, and I have to get back to writing my book, so I can't discuss them in this article. For instance, what if your Samba server is a PDC -- how will that influence the situation? What if your Samba server has the kind of password setup so that the client's changing his password changes the Samba password on the server? How does that influence things?

I obviously can't say for sure, but my educated guess would be the principles are the same. You can't fix overwritten passwords until you add the three Lanman lines to smb.conf and reboot the server. Win9x and ME clients can't log in until Lanman has been restored and the passwords have been fixed, and then only after they log out and log back in to get rid of any cached passwords. It shouldn't be rocket science.

And then there's the matter of /etc/samba/smbpasswd, which is deprecated in favor of one of the new Samba databases such as tdbsam or ldapsam. After you fix the problem, but before restoring production would be an excellent time to make this migration.