Troubleshooters.Com, T.C Linux Library and Knoppix Knowhow Present

Using Your Knoppix CD as a Rescue CD
Copyright (C) 2003 by Steve Litt

[ Linux Library | Knoppix Knowhow | Troubleshooters.Com | Email Steve Litt | Copyright Notice ]

Contents:

Executive Summary

The worst has happened. Your Linux box won't boot, and your data sits there unreachable. Luckily, you have your trusty Knoppix CD, so you can bust into the most locked up system, as long as the filesystems are still intact. Armed with your Knoppix disk, you can probably re-enable booting. But even if that's not possible, you can copy your data to a spare partition and send it, via the network, to another machine.

This document discusses the various ways you can use your Knoppix CD to bust back into a zapped machine, take control, and either re-enable booting or move your data to safety.

Disclaimer

Obviously, you use this document at your own risk. I am not responsible for any damage or injury caused by your use of this document, or caused by errors and/or omissions in this document. If that's not acceptable to you, you may not use this document. By using this document you are accepting this disclaimer.

What is Knoppix

Knoppix is a Linux distro that can run completely off a bootable CD, with no need for a hard disk. Its primary use is as a Linux advocacy tool. Give a Knoppix CD to a Windows user, tell him to boot to that CD (most Windows users are smart enough to know how to set the bios to boot a CD), and watch the fun. Within a minute Knoppix has found all his hardware, and has even mounted all his partitions read-only, in a nice, easy KDE environment. The Windows user can experiment with OpenOffice, Evolution, Mozilla, Gimp, and many other wonderful pieces of Linux software.

Like all Linux distros, when you boot to the Knoppix CD you come to a boot prompt. On high quality hardware you'd typically press Enter, but in some situations you'll need to pass boot parameters (Knoppix calls them "cheat codes") to the kernel. The most frequently needed parameters involve video, because Knoppix's default resolution might produce too-small fonts on a small monitor, or it might overdrive the monitor. For an overdriven monitor you can either set the vertical refresh rate (knoppix xvrefresh=) or the geometry (knoppix screen=). To see a list of all the possible boot parameters, press the F2 button at the boot prompt. The following is a list of the boot parameters on my Knoppix CD:

Boot param
 Purpose
knoppix lang=
 Choose language/keyboard
knoppix desktop=
 Use window manager other than KDE
knoppix screen=
 1280x1024, 1024x768, 800x600,640x480 etc
knoppix xserver=
 Use X server other than default XFree86
knoppix wheelmouse
 Use IMPS/2-compatible wheelmouse
knoppix xvrefresh=60
 Use 60Hz vertical refresh. Try this or lower values if your monitor overdrives
knoppix 2
 Textmode only (runlevel 2)
knoppix 1
 Single user mode (runlevel 1). Use this to perform rescue operations
knoppix floppyconfig
 Run "knoppix.sh" from a floppy.
knoppix no <hardware>
 Disable hardware detection on a device
knoppix blind brltty=
 Use Braille-Terminal, no X
failsafe
 (Almost) no hardware detection
expert
 Interactive startup/setup

You can combine boot parameters. For instance, you could do this:
knoppix wheelmouse xvrefresh=60 lang=us
Once you press enter on the boot prompt (whether or not you input boot parameters), a rather long boot process takes place. The process is long because Knoppix is detecting all your hardware, and readonly mounting all your partitions.

Knoppix as an Investigational Tool

Before your repair a non-booting system, you need to understand it. Rescue CD's such as the Red Hat Installer and the old Linuxcare CD's don't make such investigation easy. Knoppix does.

Knoppix' default boot is GUI mode. The desktop contains an icon for every partition in the machine. Single-clicking any of these partitions brings up its contents in a file manager so you can instantly deduce its normal mountpoint. Write down each partition device and its mountpoint so that you have the necessary information when you do the actual repair.

You might want to look at some configuration files. And naturally you'll want to browse the logs. Finally, check out the "files" in the /proc directory, especially /proc/meminfo and /proc/partitions.

If there was a question about features like sound and video, check whether they work in Knoppix. If so, copy their configuration files to a place on one of the system's partitions.

Once your investigation's complete, it's time to bust in and take control of the machine.

Backing Up from the Knoppix Environment

In many cases, your first responsibility is to protect the data on a no-boot computer. Boot to your Knoppix CD. and take measures to copy the data off the computer.

By far the easiest way is to copy the data to a CD. This is a great alternative, but there are several requirements:
  1. The computer must have a CD burner (or you must be willing to install one)
  2. The computer must have a partition, that can be mounted read/write, that has enough space to hold image files

Backing Up to CD

To back up to a CD, follow these instructions:
Assuming you have a partition you can mount writeable, you can use tar, gzip, mkisofs and cdrecord to record the CD. This has the advantage of better verification and also compression, meaning you can get more on a single CD. But it involves more thinking, and in tense situations, you might want to go the brainless route.

Network Backups

If you can't back up to CDROM, back up via the network. If necessary, install a network card in the no-boot computer, and Knoppix will find and use it. The network card should be fast (minimum 100Mbps) because of the huge data amounts. Likewise, all wiring, hubs or switches, and the network card on the destination computer should support the high speed.

There's no reason why the no-boot computer or the destination computer should be on the LAN. You could hook them together with two network cables and a hub, or even a crossover network cable (although that reduces troubleshooting test points).

The destination computer must have a partition with enough room to hold the backup, and it must have a DHCP server to give the Knoppix booted computer an address. Of course, if you're connecting to the destination computer through the LAN, any DHCP server on the LAN can give the Knoppix booted computer an address, but take care that it's in the same subnet as the destination computer.

The destination computer must have an ssh server. Or, if you're doing an NFS transfer, it must have an NFS server and a properly configured /etc/exports file.

Backing Up Via ssh

The easiest way I've found is by using ssh. Assuming the following:
In the preceding circumstances, the following command will back up the former home directory will be backed up to emergency.tgz in the /scratch directory on 192.168.100.4:
tar czf - /mnt/hda4 | ssh myuid@192.168.100.4 'cat > /scratch/emergency.tgz'
The emergency.tgz file will extract files to a mnt/hda4/ directory instead of a /home directory. If it's necessary to restore this data, a mv command can be used to put the data in the right place after extraction.

The preceding example backs up an entire partition. To back up a directory within a partition, just append the directory. For instance, had you wanted to back up /home/myuid only, the argument to the tar command would have been /mnt/hda4/myuid.

Backing Up Via NFS

So far I've not found a way, from a Knoppix booted machine, to successfully mount an exported directory, but I'll continue trying to find one. When I do, network backups will be trivial -- something like this:
tar czf emergency.tgz /mnt/nfs

Huge Backups

The backups mentioned in this article fit on a CD or a spare partition. Sometimes that isn't realistic. In such cases, I imagine the ssh method could be used to write directly to tape. I don't know how to do that, but it's probably not too difficult.

Summary

When a computer becomes unbootable, often your first responsibility is to protect the data on that computer. By booting Knoppix on the unbootable computer, you gain access to its partitions and can copy its data either to CD's, or over the network for disposition on another computer. It's often best to perform this backup action before attempting to restore the computer to bootability.

Busting Back Into a Locked Up Machine

You told yourself not to mess with lilo.conf! But nooooooo, you had to mess with it, run lilo, and reboot. And now you get nothing.

Don't worry, Knoppix to the rescue.

First step is to investigate, as described in Knoppix as an Investigational Tool. Once fully armed with information, use your Knoppix CD to bust back in to even the most locked up systems.

Place the Knoppix CD in the CD drive, shut down the system, set the bios to boot from the CD, and bring the system back up. Now, obtain a root prompt by pressing Ctrl+Alt+F2. Actually, the F2 could be anything between F1 and F4, because those are the four virtual terminals. Ctrl+Alt+F5 houses the GUI, which cannot give you a root prompt. Unlike most distributions, Ctrl+Alt+F2 in Knoppix gives you a root prompt without requiring a password.

At the root prompt, you are the root user, with full access to the system.

First, mount the root partition by its device number. For instance, if the partition mounted as the root directory was /dev/hda2, perform the following command:
mount /dev/hda2 /mnt/hda2
Note that Knoppix creates directories for each partition, under the /mnt directory, so there's no need to create /mnt/hda8. What you do next depends on whether you want to use the Knoppix programs to achieve the repairs, or whether you'd prefer to use the programs of the computer under repair. Your choice depends on many factors:
Use the Knoppix Utilities when...
 Use the underlying system's utilities when...
  • You believe the underlying system's utilities have been compromised (rootkit, etc).
  • You're performing repairs requiring the underlying system's root partition or /usr directory or other important partitions to be unmounted.
  • The Knoppix utilities are easier to use.
  • The underlying system's partitions are so damaged as to be useless in restoring bootability.
  • You find it less confusing to work within the original system's directory structure.
  • You're trying to fix a lilo problem.
  • The underlying utilities are easier to use than the Knoppix utilities.

Using the underlying System's Utilities


Assume the following mounts in the original system:

/dev/hda1
 /boot
/dev/hda2 /
/dev/hda3 /usr

Do the following:

mount /dev/hda2 /mnt/hda2
chroot /mnt/hda2
mount /dev/hda1 /boot
mount /dev/hda3 /usr
You now have a replica of the original filesystem, making creation and modification of config files MUCH easier. This is especially important when configuring lilo.conf, because many aspects of lilo.conf configuration depend on the filesystem structure when lilo is run, not when the system is subsequently booted. In other words, if, in the Knoppix world, your system had completely different mountpoints, you'd need to change those mountpoints within lilo.conf, and then change them back once the system was repaired. Ughhh!

When possible, it's often, but not always easiest using your system's native filesystem to accomplish the repairs.

Always remember to unmount everything before rebooting and attempting to boot the original system. Unmounting guarantees that everything will be written to disk. Do the following: 
umount /dev/hda3
umount /dev/hda1
exit
umount /dev/hda2
The exit command undoes the chroot performed earlier.

With the partitions unmounted, from the Knoppix command prompt, type reboot. The CD will pop out. Remove it, press Enter, and the system will attempt to boot its original operating system.

Using the Knoppix Utilities

Unfortunately, it's not always advisable or even possible to use the native system's utilities. For instance, one or more of the native system's partitions might be so damaged as to be unusable. Or maybe you suspect that you've been rootkitted and don't trust the native system's utilities.

The real showstopper is that if you boot to Knoppix and then chroot to the underlying system's root partition, the functional /proc and /dev special directories are unavailable. You might find contents in the /dev directory, but they're not real devices. Sometimes you can perform /proc and dependent tasks before chrooting. Other times it's better to byte the bullet and decline to chroot. Except in special cases such as lilo.conf, it's not difficult.

Once again, assume the following mounts in the original system:

/dev/hda1
 /boot
/dev/hda2 /
/dev/hda3 /usr

First, perform actions requiring unmounted partitions. The tune2fs, fsck and e2fsck commands come to mind. Then mount the drives:
mount /dev/hda1 /mnt/hda1
mount /dev/hda2 /mnt/hda2
mount /dev/hda3 /mnt/hda3

Now you can work on individual files and run programs. Perhaps you need to copy files onto the computer. This is particularly true if you blew a computer compile and now must get it back. Don't forget that with modular kernels, the original /lib/modules/version_number directory must be restored. Perhaps the best bet is to compile a monolithic kernel on another machine, place it and its associated initrd file on the blown machine, modify grub.conf, run grub_install, and reboot.

Always remember to unmount everything before rebooting and attempting to boot the original system. Unmounting guarantees that everything will be written to disk. Do the following: 

umount /dev/hda3
umount /dev/hda2
umount /dev/hda1

With the partitions unmounted, from the Knoppix command prompt, type reboot. The CD will pop out. Remove it, press Enter, and the system will attempt to boot its original operating system.

When Things Go Wrong

"Does not boot" problems are seldom simple. They're usually caused by system level problems -- often arcane ones. Here are some troubleshooting tips:

Summary

When things go wrong, you need Knoppix. A Knoppix CD can allow you to boot an unbootable computer, salvage and transfer its data, and then fix whatever is preventing bootup.

See also: [ Linux Library | Knoppix Knowhow | Troubleshooters.Com | Email Steve Litt | Copyright Notice ]

Copyright (C)2003 by Steve Litt. -- Legal