Troubleshooters.Com Presents

Linux Productivity Magazine

Volume 2 Issue 4, April 2003

Copyright (C) 2003 by Steve Litt. All rights reserved. Materials from guest authors copyrighted by them and licensed for perpetual use to Linux Productivity Magazine. All rights reserved to the copyright holder, except for items specifically marked otherwise (certain free software source code, GNU/GPL, etc.). All material herein provided "As-Is". User assumes all risk and responsibility for any outcome.

See also Troubleshooting Techniques of the Successful Technologist
and Rapid Learning: Secret Weapon of the Successful Technologist
by Steve Litt

[ Troubleshooters.Com | Back Issues |Troubleshooting Professional Magazine ]

The fool wonders, the wise man asks. -- Benjamin Disraeli


Editor's Desk

By Steve Litt
Every day a hundred email viruses bounce harmlessly off my Linux/Kmail email system. These are Windows viruses propagated by folks with penetrated computers. Blissfully unaware, the owners of these computers continue spreading virii and losing control of their systems or data.

Consider today's typical denial of service attack. Hundreds or thousands of computers remain compromised, for days or weeks, until the controlling cracker "pulls the trigger", causing those compromised computers to to flood the target website. What is the legal liability of those compromised computers? Would you ever want to find out first hand what that legal liability is?

Security means preventing penetration of your system, but it also means taking quick and decisive action when, despite your best precautions, your system is penetrated. The first step in taking quick and decisive action is knowing that your system has been compromised. How do you find out?

System logs are certainly handy. You can see evidence of password guessing and other suspicious activities. Logs are ideal for tracing the steps of the cracker as he penetrates. But who has the time and discipline to examine logs on a daily basis?

Penetration usually involves a change of some kind. Perhaps a new port has opened up, or a new service is running. By far the most common change is that a file has changed. If you can identify a key subset of files, and on a daily basis monitor those files for changes, you'll be in a good position to detect intrusion.

Tripwire is an Open Source program created to monitor changes in a key subset of files identified by you, and report on any changes in any of those files. When changes are detected, you, as the sysadmin, can determine whether those changes occurred due to normal, permitted activity, or whether they where caused by a breakin. If the former, you can update the system baseline to the new files. If the latter, you can shut down and begin repair and forensic activities.

Tripwire's principle is simple enough. The sysadmin identifies key files and causes Tripwire to record checksums for those files. He also puts in place a cron job to scan those files at intervals (daily or more frequently), comparing to the original checksum. Any changes, additions or deletions are reported, so the proper action can be taken.

This issue of Linux Productivity Magazine is devoted to Tripwire, which can alert you quickly when there's an intrusion. So kick back, relax, enjoy, and remember that if you're an Open Source user, this is your magazine.
Steve Litt is the author of Samba Unleashed.   Steve can be reached at Steve Litt's email address.

Help Publicize Linux Productivity Magazine

By Steve Litt
Loyal readers, I need your help.

For months I've publicized Linux Productivity Magazine, expanding it from a new magazine to a mainstay read by thousands. There's a limit to what I can do alone, but if you take one minute to help, the possibilities are boundless.

If you like this magazine, please report it to one of the Linux magazines. Tell them the URL, why you like it, and ask them to link to it.

I report it to them, but they don't take it very seriously when an author blows his own horn. When a hundred readers report the magazine, they'll sit up and take notice.

Reporting is simple enough. Just click on one of these links, and report the magazine. It will take less than 5 minutes.

News Mag
Submission URL or Email address
Just fill in the short form.
Just fill in the short form.
Linux Weekly News
Just tell them the URL, why you like it.
Just tell them the URL, why you like it.
Just fill in the short form.
Newsfactor Network
Just tell them the URL, why you like it.
The Linux Knowledge Portal
Just tell them the URL, why you like it.
OS News
Just tell them the URL, why you like it.
Only for LPM issues involving the Linux desktop, not for programming or server issues.

If you really like this magazine, please take 5 minutes to help bring it to a wider audience. Submit it to one of the preceding sites.
Steve Litt is the author of Samba Unleashed.   Steve can be reached at Steve Litt's email address.

Tripwire Overview

By Steve Litt
Tripwire isn't rocket science. It's a database of file checksums, and programs to update and report on that database. It also contains rules concerning the severity of various types of anomolies. These rules are contained in a policy file. When a report shows errors, the source of those errors must be one of three things:
The following flowchart shows, on a continuous basis, the use of Tripwire:

Flow chart of the tripwire checking and adjustment process

In summary, Tripwire detects and reports on changes in thousands of strategic system files. If a changed file is detected, you determine if the change is due to normal system activities. If so, modify the Tripwire database so the change is no longer reported. If the change is not due to normal system activities, investigate it as a potential crack.
Steve Litt is the author of "Troubleshooting Techniques of the Successful Technologist".  Steve can be reached at Steve Litt's email address .

Tripwire Installation and Initial Configuration

By Steve Litt
Tripwire installs the same as other programs -- via package management or by compilation. This article discusses Red Hat RPM installation.

Your first step is to see whether Tripwire is already installed. The following command will tell you:
rpm -q tripwire
If the preceding command says "not installed", go ahead and install it. Your Linux Install CD probably contains the Tripwire RPM. On my Red Hat 8.0 set it's on CD 3. In that case, mount the CD and perform the following command:
rpm -Uvh /mnt/steve/RedHat/RPMS/tripwire-2.3.1-14.i386.rpm
That command installs Tripwire, and also creates the all-important /etc/tripwire directory. Among the many files in this directory are three crucial ones:
High level tripwire config
Rules for various files and directories
Tripwire configuration script

Here's the process for initial Tripwire configuration:
In the preceding, substitute the appropriate time/datestamp for the "timetamp" designation in the command.

Edit twcfg.txt and twpol.txt

The twcfg.txt file contains high level config info such as filenames, processes, and various flags. It's likely to be OK as is.

The twpol.txt file is a human readable version of the rules for various files and directories. It's an excellent start. If you know of files, directories or other objects on your server that aren't included in the defaulttwpol.txt, or if you feel their policies in twpol.txt aren't appropriate, make the necessary changes.

It should be noted that twpol.txt contains many filenames which probably don't exist on your system. This creates various errors on your reports. One might be tempted to remove those filenames from twpol.txtto get rid of the errors. Don't do it. If those files later appear on the system, they can be used by crackers and won't be checked by Tripwire.

The errors produced by missing files are in a section pretty much by themselves.


This script acquires the local and site passwords, and performs other initialization. It must be run by user root. Passwords should be at least 8 characters and less than or equal to 1023 characters. Pick passwords memorable to you but to nobody else, even people who might research your life. Be sure to use punctuation or mix upper and lower case letters.

When running, you'll be prompted for the site passphrase and the local passphrase (these are passwords). Make them memorable (to you), and make them appropriately secure for the situation. Be sure to remember which is which, as you'll be repeatedly prompted for them.

/usr/sbin/tripwire --init

This command initializes the Tripwire database. It takes several minutes. You'll see several messages, including some stating "no such file or directory". Here's an example:
### Warning: File system error.
### Filename: /root/.xauth
### No such file or directory
### Continuing...

That's OK. Ultimately you're looking for lines looking something like the following:

Wrote database file: /var/lib/tripwire/litttest.domain.cxm.twd
The database was successfully generated.

The success message tells you that you've generated the database, and can continue to the next step.

/usr/sbin/tripwire --check

This command takes several minutes. It performs an integrity check to discover what files have been added, deleted or modified. The results are recorded in such a way that they can be reported on. If you want to directly capture the screen output, you can pipe it to a log file as follows:
/usr/sbin/tripwire --check | tee > /securedirectory/tripcheck.log
Make sure /securedirectory is not in the /root tree or it will be reported by Tripwire as an add or change. And of course, it must be a very secure directory so outsiders can't see this valuable security information. The /tmp directory is NOT APPROPRIATE because it's world readable and world writeable.

/usr/sbin/twprint -m r --twrfile /var/lib/tripwire/report/timestamp.twr | less

This command reports on the results of a tripwire --check command. Each tripwire --check command writes a .twr file with a specific timestamp, so substitute the appropriate timestamp. The result will be a nicely formatted report on the requested tripwire --check command.

The report contains four parts, each of which has one or more sections. The parts are as follows:

Report Summary
Lists the host name, IP address, policy file, config file, database and command used to generate the report.

Rule Summary
This part contains a summary of adds, changes and deletes, together with a line specifying the total violations found. This part helps you quickly see any problems.

Object Detail
This part contains specifics of violations reported in the Report Summary. This is how you determine whether a violation is due to normal system operation, or a possible crack.

Error Report
This part lists all sorts of errors, most of which will probably be files listed in twpol.txt that don't exist (and haven't existed on previous check runs). This section isn't as useful as the others, because of all the error messages that don't really indicate a problem.

Examine the report. Of primary interest is the "Rule Summary" and "Object Detail" sections. Ideally there are no errors, adds, changes or deletions, but usually there's at least one Tripwire specific file that changed -- typically the tripwire database (ending in .twd). As long as those changes were due to your use of Tripwire, that's fine.

Assuming the violations were due to normal system operation, you need to determine if the problem was an incorrect policy file (twpol.txt). If so, the policy must be changed. However, the problems are usually due to a user unexpectedly accessing a file. For instance, if you edit, change or delete anything in the /root directory, it will be reported as a violation (assuming the default policy file). To fix the user-caused violations, you run an update.

/usr/sbin/tripwire --update --twrfile /var/lib/tripwire/report/timestamp.twr

This command updates the database so all files have their new values. In other words, a subsequent run of tripwire --check won't report those files as violations.

Of course the tripwire database itself is an exception. Every time, a .bak file is created for it, and that .bak file is always an add.

Steve Litt is the author of Rapid Learning: Secret Weapon of the Successful Technologist. He can be reached atSteve Litt's email address.

Ongoing Tripwire Use

By Steve Litt
Your /etc/cron.daily directory should contain a file called tripwire-check or similar. That script runs tripwire every day. Depending on how you configured tripwire, you might be emailed with the results. In such cases, your role becomes a passive examination of reports, except in an unusual occurrence.
Steve Litt is the author of the Universal Troubleshooting Process courseware.   Steve can be reached atSteve Litt's email address.

The Tripwire Policy File

By Steve Litt
The purpose of the Tripwire policy file is to assign lists of things to check to various files, directories and devices. For instance, for file /etc/fstab you might want to check the file size, the md5sum, the modification date, but not the access date. That would be represented as the following rule:
/etc/fstab -> sMm-a;
Or more simply:
/etc/fstab -> sMm;
These assignments are called normal rules, whose general syntax is as follows:
object_name -> property_mask;
Sometimes the property mask is followed by a parenthesized attribute/value pair, as follows:
object_name -> property_mask (rule_attribute=value);
In Tripwire terminology, files, directories and devices are called objects, while lists of things to check are called property masks. Property masks are lists of individual properties to check, each represented by a character, as follows:

Attribute to check
Attribute to check
Access timestamp
p Permissions and file mode bits
Number of blocks allocated
r ID of device pointed to by inode
(valid only for device objects)
Inode timestamp (create/modify)
s File size
ID of device on which inode resides
t File type
File owner's group ID
u File owner's user ID
Inode number
C CRC-32 hash value
File is increasing in size (a "growing file")
H Haval hash value
Modification timestamp
M MD5 hash value
Number of links (inode reference count)
S SHA hash value

To assemble a property mask, assemble characters for each property to check. For instance, pn means check permissions_and_file_mode_bits and also number_of_links. Each letter can be preceded by a plus sign, meaning check that property, or a minus sign, which means do not check that property. No sign at all is equivalent to a plus sign, while the absense of a letter is equivalent to a minus sign. The following are equivalent:


So you might wonder why plus and minus signs are used. The answer is that variables are often used to represent common property masks, and such variables need to be adjusted. Read on...


Commonly used property masks are often represented by a variable. For instance,
Now that the variable is set, you can use it like this:
/etc -> $(SEC_INVARIANT);
The preceding statement sets the property mask of object /etc to tpug, meaning check file type, permissions and file mode bits, file owner's user ID, and file owner's group ID.

Let's say you have another directory which should have the same properties, except on this other directory it's permissible for the directory's owner to change. Also, on this new directory, the number of links must never change, because you do not want links to this directory. In this case, you could make the following rule:
/newdir -> $(SEC_INVARIANT) -u +n;
Tripwire has several predefined variables which cannot be changed:

Property Mask
$( ReadOnly)
+pinugtsdbmCM-rlacSH ReadOnly is good for files that are widely available but are intended to be read-only.
$(Dynamic) +pinugtd-srlbamcCMSH Dynamic is good for monitoring user directories and files that tend to be dynamic in behavior.
$(Growing) +pinugtdl-srbamcCMSH The Growing variable is intended for files that should only get larger.
$(Device) +pugsdr-intlbamcCMSH Device is good for devices or other files that Tripwire should not attempt to open.
$(IgnoreAll) -pinugtsdrlbamcCMS IgnoreAll tracks a file's presence or absence, but doesn't check any other properties.

+pinugtsdrbamcCMSH-l IgnoreNone turns on all properties and provides a convenient starting point for defining your own property masks.  (For example, mymask = $(IgnoreNone) -ar;). Note that the l property (growing file) is not tracked by $(IgnoreNone).

Variables can be used for things other than property masks. Variables are also used to represent part of objects, as follows:
$(TWBIN)/tripwire -> $(SEC_BIN);

Rule Attributes

A normal rule looks like this:
object_name -> property_mask;
Sometimes the normal rule needs additional information, so a parenthesized attribute/value pair is appended as follows:
object_name -> property_mask (rule_attribute=value);
The following is a real life example:
/home -> $(SEC_INVARIANT) (recurse = 0);
The (recurse = 0) rule attribute tells Tripwire to apply this rule to the directory, but to not scan anything inside the directory.

Rule Attribute Groups

Attributes can be applied to entire groups of normal rules. The syntax is as follows:
(attribute list)
rule list;

The following is a real-life example of such a group:
# Tripwire Binaries
rulename = "Tripwire Binaries",
severity = $(SIG_HI)
$(TWBIN)/siggen -> $(SEC_BIN) ;
$(TWBIN)/tripwire -> $(SEC_BIN) ;
$(TWBIN)/twadmin -> $(SEC_BIN) ;
$(TWBIN)/twprint -> $(SEC_BIN) ;
In the preceding, all four normal rules have the name "Tripwire Binaries" and have severity $(SIG_HI). Most normal rules of the default policy that ships with Tripwire are constructed with such groups.

Stop Points

Stop Points are a special kind of rule that prevent an object from being scanned by Tripwire. The syntax of a stop point is as follows:
! object_name ;
Stop points are typically used to prevent scanning of specific objects in a directory that's being scanned. For instance:
/boot                   -> $(SEC_CRIT) ;
!/boot/ ;
!/boot/module-info ;
The preceding scans all of /boot with the $(SEC_CRIT) property mask, but does not scan /boot/ or /boot/module-info.


Directives allow conditional interpretation of the policy file and certain diagnostic and debugging operations.  Their main use is to allow several boxes to share a single policy file. The syntax of a directive is as follows:
@@ directive_name [arguments]
Tripwire supports the following directives:
Designates a section of the policy file.
Allow conditional interpretation.
@@ifhost alternative evaluation.
Ends an @@ifhost.
Print a message to standard output.
Print a message to standard output and then exit.
Marks the logical end-of-file.

The following example recurses only 1 deep for hosts mydesk andmyserver, but recurses infinitely for all other hosts:
@@ifhost mydesk || myserver
/usr/sbin -> $(SEC_BIN) (recurse = 1)
/usr/sbin -> $(SEC_BIN) (recurse = -1)

Printing Out Your Tripwire Policy

Your Tripwire policy is kept in a signed, encrypted file usually called /etc/tripwire/tw.pol. If you want to see it in plain text, run the following command:
twadmin --print-polfile | less
To actually capture the file, do the following:
twadmin --print-polfile > mypolicy.txt
Be careful. The information in your policy file represents a dire security risk if it falls into the wrong hands. You should have a good reason for doing this. The main usage of this is in order to obtain a plain text policy, which you can modify in a text editor, and then use to update tw.pol.

Modifying Tripwire's Policy

By default, Tripwire's policy file is located at /etc/tripwiretw.pol. There are two ways to modify Tripwire's policy file:
  1. twadmin --create-polfile policyfile.txt
  2. tripwire --update-policy policyfile.txt
The former is used to create an initial policy file (the tripwire --init does this for you, actually). This should be done only initially, because it reinitializes the database and can lead to a cracker's modifications being included in the baseline.  The argument to the  twadmin --create-polfile method is a text file containing the policies. This text file is typically /etc/tripwire/twpol.txt. It's a security violation to have such a plain text file hanging around -- it should be used only for diagnostics and possibly for backup (in a very secure place).

The second method is the preferred way to modify an existing database. It modifys rather than rewrites the policy file. Once again, it uses a plain text policy file. Rather than keeping old copies of text policy files around, you can obtain a new text version with the following command:
twadmin --print-polfile > mypolicy.txt

Obtaining More Information

This article has just begun to scratch the surface of Tripwire policies. The main benefit of this article is that it gives you enough understanding that you can correctly interpret Tripwire man pages. The following Tripwire man pages should give you the information you need:


The Tripwire policy file is a signed and encrypted binary file which is updated via a text version of the policy. The text version can be printed at any time using the twadmin --print-polfile textfile command. The policy file contains four kinds of statements:
  2. Rules
  3. Variables
  4. Directives
Anything to the right of a hash mark (#) is a comment. Variables represent strings and can be assigned and used. There are several predefined variables that cannot be assigned, just used. Directives begin with a double at sign (@@) and facilitate conditional interpretation and debugging.

Broadly speaking, there are two types of rules:
  1. Normal Rules
  2. Stop Points
Stop points tell Tripwire not to scan a directory, file or device. Stop points begin with an exclamation point. Normal rules assign a property mask to an object (file, directory or device) using the -> syntax. Optionally, a normal rule can have a parenthesized attribute/value pair appended to it. Normal rules can be grouped, having a single list of attribute/value pairs for all grouped normal rules.

A text version of the Tripwire policy file can be printed as follows:
twadmin --print-polfile > mypolicy.txt
You can use the produced file to modify the Tripwire policy file by hand-editing the text file and then updating with the following command:
tripwire --update-policy mypolicy.txt
There may come a time when you want to delete the existing policy file and create an entirely new one. That's a security problem because it can incorporate a cracker's changes in your baseline, but who knows, there may be some reason to do this. If so, use the following syntax:
twadmin --create-polfile policyfile.txt
This article just scratches the surface. To learn more about the policy file, see the man pages for tripwire, twadmin, and twpolicy.
Steve Litt is the author of the course on the Universal Troubleshooting Process.  He can be reached at Steve Litt's email address.

Learning More About Tripwire

By Steve Litt
This month's Linux Productivity Magazine barely scratches the surface. It is probably sufficient for a small office type environment, but for any serious commercial installation you must learn much more. Luckily, this month's LPM gives you the foundation to understand more complex sources of Tripwire info. Start with the man pages:
Next, read your distro's Tripwire documentation. A locate command will show you where it is (if it's installed).
Steve Litt is the author of "Troubleshooting Techniques of the Successful Technologist".  Steve can be reached at Steve Litt's email address .

Life After Windows: The Day Knoppix Saved My Bacon

Life After Windows is a regular Linux Productivity Magazine column, by Steve Litt, bringing you observations and tips subsequent to Troubleshooters.Com's Windows to Linux conversion.
By Steve Litt
Up until last week, loss of my operating system usually meant no backup -- often necessitating dropping back a few days to my last backup. This was usually true in Windows, where I didn't have the knowledge to re-use partitions, and sometimes true in Linux -- in cases where partition reuse failed.

Last week I had a horrid little problem. Unexplicably many of my system directories changed their group and their permissions. I was able to restore the proper settings, but was worried the problem was caused by a crack. So I decided to back up my data, reformat, and reinstall my OS.

Then things got really squirrly. My normal backup scripts failed. They never failed at the same point, but they failed repeatedly to the extent that I could not get a backup. Many of the errors appeared to be disk errors. My hard disk was a 60GB IBM GXP-75. GXP-75 are so widely reputed to go bad that more than one friend has expressed surprise that mine was still running after 2 years. I bought a new 80GB Western Digital and called it "preventive maintenance". But I still had the problem of retrieving the old data. My latest previous backup was 2 days old. Oh Oh!

Fortune smiled on me in one respect. I had followed my own advice and kept all data on a second drive -- an older IBM 20GB Deskstar with a sterling reputation. So even with a thrashed system drive, if I could somehow load an OS, I could offload the data to another computer using NFS.

Did somebody say Knoppix?

I threw a Knoppix CD in the drive, rebooted, and Knoppix came up. There was no hint of intermittence. But there wasn't enough room on my Data drive to keep ISO images or any other backup. I needed to back up via the network.

No problem. My newest experimental computer has a 160GB drive. I exported an NFS share from that computer, and mounted it on Knoppix. I performed a cp -Rp on my main data tree. Twice. Then, on the experimental computer, I performed a diff --recursive command on the two copies of the tree, verifying they were identical. Next, I created .tgz backups of all data on the mounted directory.

The last step was burning CD's of my backup .tgz files. On Knoppix, I ran Xcdroast, selected the .tgz files to back up, and backed them up. The backup required 4 CD's.

Once backed up, I changed all passwords on all computers, replaced the GXP-75 with my new Western Digital 80GB, and reinstalled Mandrake. Absolutely no data loss!

Rescue CD's have always existed, but Knoppix is the first one I've used that gives me a realistic operating environment together with graphical tools to browse the hard disk, back up to CD,and mount or export NFS. I owe the crew at Knoppix a big thank you.

So if you get yourself in a pickle and need to get data off a bombed box, consider Knoppix.
Steve Litt is the author of the course on the Universal Troubleshooting Process.  He can be reached at Steve Litt's email address.

Letters to the Editor

All letters become the property of the publisher (Steve Litt), and may be edited for clarity or brevity. We especially welcome additions, clarifications, corrections or flames from vendors whose products have been reviewed in this magazine. We reserve the right to not publish letters we deem in bad taste (bad language, obscenity, hate, lewd, violence, etc.).
Submit letters to the editor to Steve Litt's email address, and be sure the subject reads "Letter to the Editor". We regret that we cannot return your letter, so please make a copy of it for future reference.

How to Submit an Article

We anticipate two to five articles per issue, with issues coming out monthly. We look for articles that pertain to the Linux or Open Source. This can be done as an essay, with humor, with a case study, or some other literary device. A Troubleshooting poem would be nice. Submissions may mention a specific product, but must be useful without the purchase of that product. Content must greatly overpower advertising. Submissions should be between 250 and 2000 words long.

Any article submitted to Linux Productivity Magazine must be licensed with the Open Publication License, which you can view at At your option you may elect the option to prohibit substantive modifications. However, in order to publish your article in Linux Productivity Magazine, you must decline the option to prohibit commercial use, because Linux Productivity Magazine is a commercial publication.

Obviously, you must be the copyright holder and must be legally able to so license the article. We do not currently pay for articles.

Troubleshooters.Com reserves the right to edit any submission for clarity or brevity, within the scope of the Open Publication License. If you elect to prohibit substantive modifications, we may elect to place editors notes outside of your material, or reject the submission, or send it back for modification. Any published article will include a two sentence description of the author, a hypertext link to his or her email, and a phone number if desired. Upon request, we will include a hypertext link, at the end of the magazine issue, to the author's website, providing that website meets the Troubleshooters.Com criteria for links and that the author's website first links to Troubleshooters.Com. Authors: please understand we can't place hyperlinks inside articles. If we did, only the first article would be read, and we can't place every article first.

Submissions should be emailed to Steve Litt's email address, with subject line Article Submission. The first paragraph of your message should read as follows (unless other arrangements are previously made in writing):

Copyright (c) 2003 by <your name>. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, version  Draft v1.0, 8 June 1999 (Available at (wordwrapped for readability at The latest version is presently available at

Open Publication License Option A [ is | is not] elected, so this document [may | may not] be modified. Option B is not elected, so this material may be published for commercial purposes.

After that paragraph, write the title, text of the article, and a two sentence description of the author.

Why not Draft v1.0, 8 June 1999 OR LATER

The Open Publication License recommends using the word "or later" to describe the version of the license. That is unacceptable for Troubleshooting Professional Magazine because we do not know the provisions of that newer version, so it makes no sense to commit to it. We all hope later versions will be better, but there's always a chance that leadership will change. We cannot take the chance that the disclaimer of warranty will be dropped in a later version. 


All trademarks are the property of their respective owners. Troubleshooters.Com(R) is a registered trademark of Steve Litt.

URLs Mentioned in this Issue