CONTENTS

Editor's Desk Is Linux Productivity Magazine Back for Good? My Backup Philosophy Activities in Rsync Assisted Backup Systems Passwords and Password Alternatives Musings on Rsync Life After Windows: What does this have to do with Linux? GNU/Linux, open source and free software Letters to the Editor How to Submit an Article Trademarks URLs Mentioned in this Issue

Editor's Desk

By Steve Litt

• Quick daily backups
• Quick daily increments that can be restored
• Burn to removeable media on the backup server instead of the work machine

Steve Litt is the author of Troubleshooting Techniques of the Successful Technologist.   Steve can be reached at his email address.

Is Linux Productivity Magazine Back for Good?

By Steve Litt

Steve Litt is the author of Manager's Guide to Technical Troubleshooting.   Steve can be reached at his email address.

My Backup Philosophy

By Steve Litt

• Predictable and Trustworthy
• Accurate and Complete
• Restorable
• Easy to use
• Part of a good backup system

Ease of Use

Data Grows

There's a Better Way

My Philosophical Change

1. Incrementally back up over the network to update a "mirror"
2. Burn to DVD from the "mirror"

Steve Litt is the author of the Twenty Eight Tales of Troubleshooting.   Steve can be reached at his email address.

Activities in Rsync Assisted Backup Systems

By Steve Litt

1. Pulling changed files from the workstation to the backup server.
2. Using hardlinks to create "incremental backups".
3. Creating tarballs from the backup server's directory mirror.
4. Recording backup metadata.
5. Burning the tarballs onto removable media.
6. After burn verification

Pulling changed files from the workstation to the backup server.

RSYNC_RSH="ssh -c arcfour -o Compression=no -x"
rsync -vaHx --progress --numeric-ids --delete  \
    --exclude-from=pbup_backup.excludes  --delete-excluded \
    slitt@192.168.100.2:/scratch/pictures/ /stevebup/rsync/pictures/

Using hardlinks to create "incremental backups"

#!/bin/bash

function showfile()
  {
  echo -n $1
  echo -n ': '
  cat $1
  echo
  }

rm -f first.txt
rm -f second.txt

echo -n first > first.txt
ln first.txt second.txt
showfile first.txt
showfile second.txt
ls -i -1 first.txt second.txt
echo
echo -n ': append to first' >> first.txt
showfile first.txt
showfile second.txt
ls -i -1 first.txt second.txt
echo
echo -n '-- append to second' >> second.txt
showfile first.txt
showfile second.txt
ls -i -1 first.txt second.txt
echo
echo -n ', redirect to first' > first.txt
showfile first.txt
showfile second.txt
ls -i -1 first.txt second.txt
echo
rm -f first.txt
echo -n 'Brand new first' > first.txt
showfile first.txt
showfile second.txt
ls -i -1 first.txt second.txt
echo

rm -f first.txt
rm -f second.txt

#!/bin/bash
datestamp=$(head -n1 /stevebup/rsync/meta/timestamp.txt)
incname=inc_$datestamp

cd /stevebup/
echo "PLEASE WAIT, MAKING INCREMENTAL COPY, cp -al rsync $incname..."
cp -al rsync $incname

Creating tarballs from the backup server's directory mirror

• By placing filenames in the archive file instead of the media directory, I get around any OS dependencies on filenames.
• Slightly better compression than compressing every single file.
• Ability to compare the archive against a .md5 file, to verify the integrity of the backup years after creation.
• Eliminate problems with Linux' "read ahead bug" affecting CD and DVD writes.

• Ability to create .md5 and .lst files (.lst files list the files in the archive).
• Ability to log the results.
• Ability to compare the resulting tgz against its hard disk source, and error out on failure. I've never had a failure on .tgz creation, but there's always a first time. The error mechanism prevents unknowing recording of bad backups.
• Intelligent error handling.

#!/usr/bin/ruby

require 'date'

$tgzdir = "/stevebup/tgz/"
$rsyncdir = "/stevebup/rsync/"


$tgzs =	[
	["d", "d/"],
	["sl", "slitt/"],
	["p", "pictures/"],
	["a", "a3b3/"],
	["ca", "classic/a"],
	["i", "inst/"],
	["l", "tclogs/"],
	["me", "meta/"],
	]


class Logger
	attr_reader :logfname
	attr_accessor :logfile
	attr_accessor :stagename
	attr_accessor :errors
	def initialize(logfname)
		@logfname = logfname
		@errors = 0
		@logfile = File.new(@logfname, "w")
	end

	def begins()
		puts "Begin #{stagename}..."
		@logfile.puts "Begin #{stagename}..."
	end

	def success()
		puts "#{stagename} completed successfully!"
		@logfile.puts "#{stagename} completed successfully!"
	end

	def failure(errmsg)
		puts "#{stagename} failed: #{errmsg}"
		@logfile.puts "#{stagename} failed: #{errmsg}"
		errors += 1
	end

	def skipline(msg)
		puts 
		@logfile.puts
		if (msg != nil) and (msg != "")
			puts msg
			logfile.puts msg
		end
	end

end

def todaystring()
	d = Date.today()
	return zerofill(d.year - 2000) + zerofill(d.month) + zerofill(d.mday)
end


def zerofill(number)
	number += 10000
	string = number.to_s
	twodigit = string[-2, 2]
	return twodigit
end

def tar_cre_string(abbrev, dir)

	command = "tar czvf #{$tgzdir}#{abbrev}#{$datestring}.tgz #{dir}"
	return command
end

def tar_diff_string(abbrev, dir)

	command = "tar dzvf #{$tgzdir}#{abbrev}#{$datestring}.tgz"
	return command
end

def tar_md5_string(abbrev)
	command = "md5sum #{$tgzdir}#{abbrev}#{$datestring}.tgz > #{$tgzdir}#{abbrev}#{$datestring}.md5"
	return command
end

def tar_lst_string(abbrev)
	command = "tar tzvf #{$tgzdir}#{abbrev}#{$datestring}.tgz "
	command = command + "| sed -e \"s/^.* //\" | "
	command = command + "sort > #{$tgzdir}#{abbrev}#{$datestring}.lst"
	puts command
	return command
end

def do1tgz_string_only(commands, abbrev, dir)
	commands.push(tar_cre_string(abbrev, dir))
	commands.push(tar_diff_string(abbrev, dir))
	commands.push(tar_md5_string(abbrev))
	commands.push(tar_lst_string(abbrev))
end

def do1tgz(tgzlogger, tasklogger, abbrev, dir)
	this_tgz_errors = 0
	tgzlogger.stagename ="Directory #{dir} (#{abbrev}{$datestring}.tgz)"
	tgzlogger.skipline("")
	tgzlogger.begins()
	

	tasklogger.skipline("Directory #{dir} as #{abbrev}{$datestring}.tgz")
	tasklogger.stagename = ("Creating #{$tgzdir}#{abbrev}#{$datestring}.tgz")
	tasklogger.begins()
	cre_return = system(tar_cre_string(abbrev, dir))
	if cre_return then
		tasklogger.success()
	else
		tasklogger.failure("")
		this_tgz_errors += 1
	end

	tasklogger.stagename = ("Diffing #{$tgzdir}#{abbrev}#{$datestring}.tgz")
	tasklogger.begins()
	diff_return = system(tar_diff_string(abbrev, dir))
	if diff_return then
		tasklogger.success()
	else
		tasklogger.failure("")
		this_tgz_errors += 2
	end

	
	tasklogger.stagename = ("Creating md5 #{$tgzdir}#{abbrev}#{$datestring}.md5")
	tasklogger.begins()
	md5_return = system(tar_md5_string(abbrev))
	if md5_return then
		tasklogger.success()
	else
		tasklogger.failure("")
		this_tgz_errors += 4
	end

	tasklogger.stagename = ("Creating lst #{$tgzdir}#{abbrev}#{$datestring}.lst")
	tasklogger.begins()
	lst_return = system(tar_lst_string(abbrev))
	if lst_return then
		tasklogger.success()
	else
		tasklogger.failure("")
		this_tgz_errors += 8
	end

	if this_tgz_errors == 0 then
		tgzlogger.success()
	else
		errmsg = "failed on step(s) " 
		if this_tgz_errors % 1 == 1 then
			errmsg += "(CREATE) "
		end

		this_tgz_errors /= 2

		if this_tgz_errors % 1 == 1 then
			errmsg += "(DIFF) "
		end

		this_tgz_errors /= 2

		if this_tgz_errors % 1 == 1 then
			errmsg += "(MD5) "
		end

		this_tgz_errors /= 2

		if this_tgz_errors % 1 == 1 then
			errmsg += "(LST) "
		end

		tgzlogger.failure(errmsg)
	end

end

def main()
	$datestring = todaystring()
	Dir.chdir($rsyncdir)
	system("pwd")
	system("sleep 1")
	tasklogger = Logger.new(ENV['HOME'] +"/maketgz_task.log")
	tgzlogger = Logger.new(ENV['HOME'] + "/maketgz_tgz.log")
	$tgzs.each do |tgz|
		do1tgz(tgzlogger, tasklogger, tgz[0], tgz[1])
	end
end

main()

Recording backup metadata

#!/bin/bash
rm -rf /stevebup/rsync/meta/*
date +%Y%m%d_%H%M%S > /stevebup/rsync/meta/timestamp.txt
cp -Rp /d/bats/backup /stevebup/rsync/meta/

Burning the tarballs onto removable media

After-burn verification

#!/bin/bash

echo
echo

mountpoint=/mnt/cdrom

echo Mountpoint =$mountpoint=

logfile=$HOME/verifycd.tmp
rm -f $logfile

for tgz in $mountpoint/*.tgz
do
	md5=${tgz//\.tgz/\.md5}
	echo -n "Comparing Checksums for $tgz & $md5, please wait... "
	md5val=`cut -d " " -f 1 $md5`
	tgzval=`md5sum $tgz | cut -d " " -f 1`
	echo " finished."
	echo "$md5 value==>$md5val<=="
	echo "$tgz value==>$tgzval<=="
	if (test "$md5val" = "$tgzval"); then
		echo $tgz is good. >> $logfile
	else
		echo ""
		echo $tgz MD5SUM MISMATCH! ERROR ERROR ERROR ERROR ERROR! >> $logfile
	fi
	echo
done
cat $logfile

Steve Litt is the author of the Universal Troubleshooting Process Courseware.   Steve can be reached at his email address.

Passwords and Password Alternatives

By Steve Litt

Steve Litt is the author of Rapid Learning: Secret Weapon of the Successful Technologist.   Steve can be reached at his email address.

Musings on Rsync

By Steve Litt

Steve Litt is the author of Samba Unleashed.   Steve can be reached at his email address.

Life After Windows: What does this have to do with Linux?

Life After Windows is a regular Linux Productivity Magazine column, by Steve Litt, bringing you observations and tips subsequent to Troubleshooters.Com's Windows to Linux conversion.

By Steve Litt

Steve Litt is the founder and acting president of Greater Orlando Linux User Group (GoLUG).   Steve can be reached at his email address.

GNU/Linux, open source and free software

By Steve Litt

"GNU/Linux" is probably the most accurate moniker one can give to this operating system. Please be aware that in all of Troubleshooters.Com, when I say "Linux" I really mean "GNU/Linux". I completely believe that without the GNU project, without the GNU Manifesto and the GNU/GPL license it spawned, the operating system the press calls "Linux" never would have happened.

I'm part of the press and there are times when it's easier to say "Linux" than explain to certain audiences that "GNU/Linux" is the same as what the press calls "Linux". So I abbreviate. Additionally, I abbreviate in the same way one might abbreviate the name of a multi-partner law firm. But make no mistake about it. In any article in Troubleshooting Professional Magazine, in the whole of Troubleshooters.Com, and even in the technical books I write, when I say "Linux", I mean "GNU/Linux".

Steve Litt has used GNU/Linux since 1998, and written about it since 1999. Steve can be reached at his email address.

Letters to the Editor

All letters become the property of the publisher (Steve Litt), and may be edited for clarity or brevity. We especially welcome additions, clarifications, corrections or flames from vendors whose products have been reviewed in this magazine. We reserve the right to not publish letters we deem in bad taste (bad language, obscenity, hate, lewd, violence, etc.).

Submit letters to the editor to Steve Litt's email address, and be sure the subject reads "Letter to the Editor". We regret that we cannot return your letter, so please make a copy of it for future reference.

How to Submit an Article

Any article submitted to Linux Productivity Magazine must be licensed with the Open Publication License, which you can view at http://opencontent.org/openpub/. At your option you may elect the option to prohibit substantive modifications. However, in order to publish your article in Linux Productivity Magazine, you must decline the option to prohibit commercial use, because Linux Productivity Magazine is a commercial publication.

Obviously, you must be the copyright holder and must be legally able to so license the article. We do not currently pay for articles.

Troubleshooters.Com reserves the right to edit any submission for clarity or brevity, within the scope of the Open Publication License. If you elect to prohibit substantive modifications, we may elect to place editors notes outside of your material, or reject the submission, or send it back for modification. Any published article will include a two sentence description of the author, a hypertext link to his or her email, and a phone number if desired. Upon request, we will include a hypertext link, at the end of the magazine issue, to the author's website, providing that website meets the Troubleshooters.Com criteria for links and that the author's website first links to Troubleshooters.Com. Authors: please understand we can't place hyperlinks inside articles. If we did, only the first article would be read, and we can't place every article first.

Submissions should be emailed to Steve Litt's email address, with subject line Article Submission. The first paragraph of your message should read as follows (unless other arrangements are previously made in writing):

Copyright (c) 2003 by <your name>. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, version  Draft v1.0, 8 June 1999 (Available at http://www.troubleshooters.com/openpub04.txt/ (wordwrapped for readability at http://www.troubleshooters.com/openpub04_wrapped.txt). The latest version is presently available at  http://www.opencontent.org/openpub/).

Open Publication License Option A [ is | is not] elected, so this document [may | may not] be modified. Option B is not elected, so this material may be published for commercial purposes.

After that paragraph, write the title, text of the article, and a two sentence description of the author.

Why not Draft v1.0, 8 June 1999 OR LATER

Trademarks

URLs Mentioned in this Issue

• http://www.sanitarium.net/golug/rsync_backups.html: Kevin Korb's outstanding "Backups using rsync" presentation.
• http://www.troubleshooters.com: Steve Litt's Troubleshooting website.
• http://www.troubleshooters.com/bookstore/ttech.htm: Home of the book "Troubleshooting Tools of the Successful Technologist.
• http://www.samba.org/rsync/: Home page of the Rsync free software project.
• http://www.samba.org/: Home page of the Samba free software project.
• http://www.jdmz.net/ssh/.: How to run Rsync without passwords, and do it with as much safety as possible.
• http://samba.org/~tridge/phd_thesis.pdf: This is the Ph.D. thesis of Andrew Tridgell, which includes three chapters on Rsync. It's fascinating.
• http://www.backup-software-reviews.com/: Backup Software Reviews -- A review of 18 backup programs for Windows.
• http://www.grsoftware.net/backup/software/grbackpro.html: Website for the GRBackPro backup software.
• http://www.zipbackup.com/: Home page for the ZipBackup software, which appears to me to be a user friendly backup-centric front end to PKzip or Winzip.
• http://www.backup-software-reviews.com/: A review of 18 Windows backup programs.

_