Troubleshooters.Com Presents

Troubleshooting Professional Magazine

Volume 3, Issue 2, February 1999
When the Going Gets Tough

Copyright (C) 1998 by Steve Litt. All rights reserved. Materials from guest authors copyrighted by them and licensed for perpetual use to Troubleshooting Professional Magazine. All rights reserved to the copyright holder, except for items specifically marked otherwise (certain free software source code, GNU/GPL, etc.). All material herein provided "As-Is". User assumes all risk and responsibility for any outcome.

[ Troubleshooters.Com | Back Issues ]

Contents

Editors Desk
The Last Mile
When the Going Gets Tough
Escalation Procedures
Cooperation
Refusal of Unprofitable Repairs
Delay of Repair
Troubleshooter Slave Labor
Linux Log
Letters to the Editor
How to Submit an Article
URLs Mentioned in this Issue

Editors Desk

By Steve Litt
How do you tell the difference between ninja Troubleshooters and so-so Troubleshooters?  On 90% of repairs there's no way. Both have the necessary skills for all but the toughest 10% of repairs. But that last 10% makes the difference, doesn't it. With increasing use of the valid Troubleshooting processes, the quality of a Troubleshooter is increasingly determined by how he or she handles the nastiest problems.

This issue of Troubleshooting Professional Magazine is devoted to handling the nasty problems. So kick back and relax as you read this issue. And remember, if you're a Troubleshooter, this is your magazine. Enjoy!

Steve Litt can be reached at Steve Litt's email address.


The Last Mile

By Steve Litt
Most Troubleshooting is pretty easy. The 1982 Buick that intermittently bucks, hesitates and stalls at freeway speeds. The car that won't start and there's an inch of corrosion around the battery terminals. The Linux software that runs for user root but not for user myuid. If all problems were like these, we'd be getting minimum wage. And indeed, for all of us but Troubleshooting Hired Guns (the guy who comes in after the local guys give up), 30% to 70% of the work is no-brainer. Probably 1 in 10 repairs offers real challenge. Sometimes they take twice as long, sometimes they take 100 times as long. Intermittents, problem gets out of the box, inadequate test points, inadequate system information, low quality system -- you know the type.

That tenth repair can be thought of as "the last mile". Like the 1 mile mountain path that takes longer to walk than the 10 mile flat sidewalk, the last mile of Troubleshooting wields extraordinary power over our productivity.

Imagine the normal repair takes 30 minutes, but 1 in 10 takes 10 hours. In a 40 hour week you'll do a little over 27 repairs. [ The Math Behind ]

Improve performance by doing the easy ones in half the time.

Normal repair takes 15 minutes, but 1 in 10 takes 10 hours. In a 40 hour week you'll do a little over 32 repairs, an 18% total improvement yielded from a 100% improvement of normal repairs. [ The Math Behind ]

Now instead improve performance by doing the tough ones in half the time.

Normal repair takes 30 minutes, but 1 in 10 takes 5 hours. In a 40 hour week you'll do a little over 42 repairs, an 52% total improvement over the original, yielded from a 100% improvement of tough repairs. Note also that this outcome is 29% more productive than the one produced by doubling productivity on the easy ones. Clearly the tough repairs are the bottleneck in any Troubleshooter's life, and the quicker and easier we can do them the better our lives will be. [ The Math Behind ]

So the only question left is, how will we reduce the difficulty of the last mile problems? Here are some alternatives, listed in order of Troubleshooter control:

  1. Troubleshooter Initiative
  2. Delay of repair
  3. Refusal of unprofitable repairs
  4. Escalation procedures
  5. Troubleshooter slave labor
These are listed in order of Troubleshooter control. Alternative 1 is completely in control of the Troubleshooter, while #5 is at the whim of employer policy.

Troubleshooter Initiative relies on that old cliche, "when the going gets tough, the tough get going".

Read on...
 

Steve Litt is president of American Troublebusters and Troubleshooters.Com, and editor of Troubleshooting Professional Magazine. He's also an application developer and technical writer. He can be reached at Steve Litt's email address.


When the Going Gets Tough

By Steve Litt
the tough ask "How can I narrow it down just one more time?". This is the entire basis of Troubleshooter initiative for tough problems. It's the basic, guiding principle behind quickly and easily navigating the last mile.

A simple enough question, but the effect of its asking is profound. It keeps our minds on the job at hand, preventing panic, circular thinking, finger pointing and rash decisions. We like to think of ourselves as robots when Troubleshooting. In fact we're not. Our judgement is affected by frustration, anger, economic pressure, peer pressure. The best way to minimize negative effects of those emotions is to keep our minds on the narrowing process, and the best way to do that is to ask this question.

"How can I narrow it down just one more time?" It's a one-size-fits-all question whose answers are legion. Read on...

Find one more test

This is the usual answer. Ruling out an additional section often unmasks the remainder of the correct Troubleshooting path. Any test which rules out an additional area is a valid move (within time and risk limitations). Even a "hail Mary" test is valid as long as it rules out something, and as long as all assumptions are verified.
 

Troubleshoot our Troubleshooting

"How can I narrow it down just one more time?"  Isn't that a lot like asking "How can I make my computer see the network?" It's a Troubleshooting question, and sometimes the answer isn't obvious. Time to Troubleshoot our Troubleshooting.

Exactly what is preventing further narrowing?

Get more information

The Universal Troubleshooting Process may be system independent, but knowledge of the system is still necessary. Simple problems can be solved without a manual, tough ones can't. Some manuals are useless, and supplements must be found. Occasionally our subject matter knowledge itself is insufficient -- we must learn more.

Get the manual, or a book, or find a website, or contact others via email. Find an appropriate newsgroup or mailing list. Call a friend. Ask co-workers for help.

Get better tools

When it seems like you're troubleshooting a black box, or there aren't enough test points, or assembly/disassembly is too time consuming to promote effective narrowing down, it's time to get better tools. Here are a few examples: You can get tools from vendors, or off the net, or you can make them yourself (you know, that little coathanger hook you use to put belts on pulleys, or that little debugging routine).

Adjust your attitude

When the problem boils down to lost rationality, it's time to adjust your attitude. Here are some quick tips:

Compensate for Management

Every managment has their little quirks, and sometimes they get in the way of Troubleshooting. You could always confront managment with the facts, but sometimes that backfires on you. Unless there are safety or security concerns, sometimes the best policy is to quietly violate policy. Of course, sometimes that can backfire on you too. Tailor your managment compensation to match your management.

Summary

We all know there's no magic bullet for Troubleshooting. No expert system, software product, diagnostic tool, troubleshooting process or smart manual. The closest thing to a Troubleshooting magic bullet is a parody of a common cliche, and if you remember nothing else from this issue of Troubleshooting Professional, please remember this:
 
When the going gets tough, the tough ask "how can I narrow it just one more time".
Steve Litt can be reached at Steve Litt's email address.
 

Escalation Procedures

By Steve Litt
It's no secret I'm not a fan of escalation. Sure, it can be a valid Troubleshooting tool. But most often it's misused to cure a symptom rather than a root cause.

Escalation is the practice of a stumped technician hand the problem over to a more experienced technician. Makes sense, right? Nobody wants to waste their time with a guy who's over his head and can't possibly solve the problem. So what's wrong with escalation?

Misuse. Too many departments use an escalation policy to hire minimum wage clerks to talk to the customer, escalating over 50% of the problems. Often it takes a few levels of escalation to find the guy who can solve the problem. The customer has to tell his story several times to several different people, be kept on hold several times, repeat the same series of tests several times, and often have his intelligence questioned several times.

It costs more than customer satisfaction (as if that isn't enough reason to stop the practice). The department ends up paying a fleet of employees whose primary purpose is to waste the customer's time. And their top-tier Troubleshooters don't accomplish much -- they need to retrace earlier steps -- but with an irate customer.

What has happened, of course, is management fixed the symptom instead of the root cause. The symptom is customer busy signals and on-holds, often caused by excessive time-to-solution. The root cause could be a complex product, or insufficient tools for the Troubleshooters, insufficient Troubleshooter experience or ability, or maybe too few Troubleshooters. So management coathangers the symptom by hiring clerks. And like all coathanger fixes, it creates problems of its own.

Now the phone is answered instantly. Great! A warm, caring person listens to the complaint. And puts you on hold! Ten minutes later a slightly less warm and caring person answers, asking you to repeat the complaint. Ten minutes later he puts you on hold. Ten minutes of on-hold music and a harried manager answers, and once again the complaint is regurgitated. Now you're on hold for forty five minutes, til a downright angry developer answers the phone, takes the complaint for the fourth time, and tells you he's fixed that problem in the next release.

Of course management's best course would be to get rid of the useless first and second line people, load up on people of the caliber of the manager, and make the developers provide documentation so those people could do their job. Cut customer time by a factor of 10, help line time (customer time minus customer on-hold time) by a factor of four.

Which is better -- Frank and Joe for $5.00 per hour each, or Fenton, whose productivity is the sum of Frank and  Joe's, for $10.00 per hour? Salary per repair is identical, but Frank and Joe require double desks, double phone lines, double computers, double square footage. The more profitable move is to hire Fenton for $10.00. Better still, hire Hunter, whose productivity is double Fenton's, for $20.00.

Once the first tier is staffed by efficient people who can self-solve 95% of the problems, hire a couple heavy hitters (Quincy and Matlock) to help out with the 5% that can't be solved on initial contact.

Steve Litt is president of American Troublebusters and Troubleshooters.Com, and editor of Troubleshooting Professional Magazine. He's also an application developer and technical writer. He can be reached at Steve Litt's email address.

Cooperation

By Steve Litt
Without escalation, Team Troubleshooting is out. Right? Wrong -- there's an excellent alternative used in almost every shop and department -- cooperation. The difference is that ownership isn't transferred.

Multi-owner problems are kind of like multi-owner cars -- likely to be lemons. With cooperation, the starting tech keeps ownership of the problem, but briefly invites a specialist in to help with a particularly thorny obstacle. Once that obstacle has been cleared, the original tech continues narrowing the problem while the specialist goes on to something else -- maybe assisting someone else.

Cooperation often involves peers rather than a junior-senior relationship. Many times Troubleshooters will hit a wall, and invite peers to come in and help narrow the problem. Thus, every Troubleshooter in the department has the expertise of every other Troubleshooter -- a situation more than a match for the toughest problems.

Another form of cooperation is the beneficial trade. Here, the problem ownership is transferred, but it's the Troubleshooters involved who make the decision, often without bothering the customer and usually to the benefit of everyone.

We traded at Pacific Stereo. I have huge, clumsy, almost useless  hands, but I'm lightning diagnosing electronic problems. Another tech was just an average diagnostician, but he could yank out a pulley, cut a 1 millimeter strip of tape, wrap it around the edge of the pulley, and put it back in about a minute (it would have taken me a half hour). So he took the mechanical problems, I took the electronic ones, and we both made more money (as did the shop).

Cooperation is the capitalism of repair. When Troubleshooters are left to their own devices, paid properly, and assuming a proper amount of work comes into the shop, they will always use cooperation to work faster, easier, and more profitably.
 

Steve Litt is president of American Troublebusters and Troubleshooters.Com, and editor of Troubleshooting Professional Magazine. He's also an application developer and technical writer. He can be reached at Steve Litt's email address.


Refusal of Unprofitable Repairs

By Steve Litt
Even the best Troubleshooter can burn out if fed a steady diet of irreparable junk. I'm a big fan of refusing unprofitable work, or giving it back, without charge, in the same condition as brought in, if it is found to be unprofitable to repair after work has begun. That is, in any case where it's ethical to do so. There are some situations where it's not ethical: When it's ethical, it's absolutely vital that unprofitable repairs be declined. Failure to do so either forces the Troubleshooter to lose money, or makes the customer pay an exorbitant price for a system that's obsolete or likely to break again.
Steve Litt is president of American Troublebusters and Troubleshooters.Com, and editor of Troubleshooting Professional Magazine. He's also an application developer and technical writer. He can be reached at Steve Litt's email address.

Delay of Repair

By Steve Litt
A good Troubleshooter knows when he or she has lost The Attitude. At that point Troubleshooting should stop, until The Attitude can be regained. This is ESPECIALLY important in the face of anger. Angry Troubleshooters are ten times more likely to accidentally cause further damage. The prudent Troubleshooter will cease repair on a system that's made him angry, frustrated, anxious or exhausted. I recommend one of the following: Unfortunately, most managements aren't familiar enough with Troubleshooting to understand the need to delay a repair to regain The Attitude. When you're sick you need a doctor's note to prove it. Here's a "Troubleshooters note" to give your boss if you need to put a problem aside to regain The Attitude:
 
Note to Employer

From: Steve Litt, Troubleshooting Process Analyst

To:__________________________________
 

Dear Sir or Madam,

Your employee, _________________, has requested to put aside troubleshooting system _________________________, citing this/these reason(s) (check one or more):

  • Fatigue
  • Anger/frustration
  • Pressure
While I am not familiar with this particular case, I generally suggest that in order to maximize profit, you grant such employee requests. 
  • Tired Troubleshooters are often ineffective on tough problems, increasing time to solution severalfold. Relegating tired Troubleshooters to "tough it out" often makes them frustrated and angry. It's not a choice, it's simply a universal human reaction. The most likely outcome of a policy of forcing the tired Troubleshooter to "keep at it" is increased turnaround time -- precicely the opposite of its intended outcome.
  • Angry Troubleshooters represent a tenfold increase in likelihood of further damage to the system. Angry Troubleshooters are not prudent, often circumventing safety procedures such as backup, current limiting, partial reassembly, and even personal safety procedures. The most likely outcome of a policy forcing angry Troubleshooters to "keep at it" is payments for damage caused and workers comp claims. The damaged equipment will increase time to solution several-fold, and increase the cost of repair even more.
  • Pressured Troubleshooters lose their rationality, engage in circular and superstitious thinking, and generally become ineffective. The pressure can come from peers, supervisors, or the employees knowledge of the consequences of a failed or slow solution. The most likely outcome of a policy of forcing pressured Troubleshooters to "keep at it" is increased turnaround time. Additionally, such pressure is contageous, often spreading throughout departments if the pressured Troubleshooters are not allowed to temporarily put aside work.


Employee's Responsibility

By submitting this request, the employee swears that he really needs this break to regain his Troubleshooting Attitude, and that when that is regained he or she will once again begin work on the problem that was put aside.
Steve Litt is president of American Troublebusters and Troubleshooters.Com, and editor of Troubleshooting Professional Magazine. He's also an application developer and technical writer. He can be reached at Steve Litt's email address.


Troubleshooter Slave Labor

By Steve Litt
Some managements try to go the last mile on Troubleshooter Slave Labor. Pressure, long hours, 24/7 on-call, unpaid overtime -- anything for a cheap solution. And what do they get? The most expensive solution. Huge training costs associated with high turnover. Terrible troubleshooting from low morale, exhaustion, and bad attitudes. But sometimes the management bungles are accidental and more subtle...

The Floater

I was a "floater" At Pacific Stereo. Floaters were the fast techs who could walk into a swamped shop, work a couple days, and clear the backlog. Since Pacific Stereo techs were on commission, floaters made big bucks.

I'll never forget an Orange County shop. A service manager and two very tired looking techs. As I walked in the Service manager said to his techs, "You watch this guy and take lessons." Quite an ego boost til I saw what was going on.

Often the service manager wasn't there, requiring the techs to wait the counter instead of fixing equipment. When the service manager was there it was even worse because he took in irreparable junk. Of course my requirement for coming to his shop was that I be able to "cherry pick" the repairs, so I skipped the junk. His techs couldn't. Then there was the rule. The rule stated that at the completion of each repair the tech must phone the customer. Customer not home? Try again later. And with a turnaround time of a week, there were constant calls from irate customers. When his techs weren't waiting the counter, playing phone tag or working on irreparable units, they could make money and satisfy the customer by doing their job.

I fixed a career high of about 15 units per day the two days I was there. That was 3 times average production. I got to know those two techs. And I can tell you they were every bit as good as me...

--*--

Compare that to the Santa Monica shop. Once again a service manager and two techs, but the techs looked happy and well fed. The service manager waited the counter, taking in only units profitable to repair. He told the customer "call us at 6pm, it will probably be ready". Or if it was close to closing time, "call us tomorrow at 6pm". He'd write the paperwork and put it on a shelf right in the repair area.

That shelf *never* filled up. After finishing a unit, each tech would go to the "in" shelf and select *the easiest* unit to fix, fix it, and put it back in the "fixed" shelves. In the inevitable lulls, they'd do the harder fixes. No outgoing phone calls. Except when units needed parts, turnaround was about 4 hours. Each tech did 8-10 units per day (that was better than my average). And you know what? I think I was as good as they were.

The service manager must have thought so too. When one of the techs quit, he called me and offered me a job at his shop, which was the best tech job in all of Pacific Stereo. Unfortunately, by that time I was a computer programmer so I had to decline...

Helping Management See the Light

If management policies are creating a problem, the first step is to see if they're necessary for some reason. Example: some safety critical environments require the Troubleshooter to submit an exact diagnostic itinerary for approval before making any tests or measurements. While this would be silly troubleshooting a departmental server, it definitely makes sense troubleshooting our nuclear defense system.

Another example: You're the PC tech and management won't let you use hard disks you know to be faster, better, and more reliable. At first sounding silly, this starts making sense when you realize there are 10 PC techs, and 1000 PC's to take care of, and you don't want to train each of 10 techs on each hard disk that leapfrogs the previous.

Once you've determined there's no good reason for the policy, nicely approach them with suggestions for streamlining. In the floater example above, I'd start with the rule that they call customers after each repair, and with the fact that they were promiscuous in their acceptance of work for repair. I'd point out that with better performance and turnaround, the customer increase would more than make up for the work they were turning down. The shop would make more money. Once that worked, I'd go on to the tougher sells like the fact that the manager was out a lot, and the policy that equipment must be repaired in the same order as brought in (that policy usually slows throughput and profit).

There is None So Blind, As He Who Will Not See

If the management policies interfere with Troubleshooting, those policies aren't required for safety or other good reasons, and management repeatedly refuses to listen to nicely made improvement suggestions -- well, the market for Troubleshooters has never been better. We can all use a raise now and then.
Steve Litt is president of American Troublebusters and Troubleshooters.Com, and editor of Troubleshooting Professional Magazine. He's also an application developer and technical writer. He can be reached at Steve Litt's email address.



Linux Log

Linux Log is now a regular column in Troubleshooting Professional Magazine, authored by Steve Litt. Each month we'll explore a facet of Linux as it relates to that month's theme. Today we'll discuss Linux from the point of view of intermittence.
When the going gets tough, the tough ask "how can I narrow it just one more time". In Linux, as often as not, the answer to that question is "get more information". That's because of its Unix roots.

The Unix crowd was macho. "Your IQ is below 140? What are you doing in Unix? You've been using Unix for less than 4 years? Why aren't you still an operator?". Where DOS and Windows had nice help files, Unix had "man pages". A man page shows the syntax of a command, usually including 30 options, some upper case and some lower case. It then goes on to briefly describe each option. That's it. Typically there are **NO EXAMPLES**. "Hey -- if you can't figure it out..."

Linux, unfortunately, still puts its primary documentation in man pages. Some have been ported to something called "info", but all too often there are still no examples. How can a normal person be expected to read a description with no examples and know what to do? Linux is a powerful, stable and inexpensive operating system, but it needs much work on documentation.

Distribution HTML

Fortunately, there are other sources. The Linux Documentation Project is mirrored on many places on the web. It has great examples. You can find it by putting 
+"linux documentation project"
in a search engine. And better still, many distributions of Linux come with select html pages from the Linux Documentation Project, useful even when not 'net connected. Here are some places to look for html docs on your machine: Obviously, the directories and contents may be different on your distribution, but this is what I got on the RedHat 5.1 distro.

Join a LUG

Join a Linux User Group. I constantly get help from ELUG, my local Linux User Group here in central Florida. And I constantly give help to others. When Linux gains corporational correctness (probably within a year), we ELUG members will be the technological elite here.

When you're in a LUG, no mystery stays unsolved for long. Knowledge dwarfs the sum of the individuals. We Linux people might not have great help files, but our User Groups are out of this world.

Web Sites, Newsgroups, Forums and Mailing Lists

I won't list all the sites. Just put the relevant words in a search engine, and go to work. You can find *anything* concerning Linux within an hour. Be sure to join the newsgroup or mailing list for your LUG.

__*__*__*__

When the going gets tough, the tough ask "how can I narrow it just one more time". We Linux people have made finding the answer into an art form.

Steve Litt can be reached at Steve Litt's email address.



Letters to the Editor

All letters become the property of the publisher (Steve Litt), and may be edited for clarity or brevity. We especially welcome additions, clarifications, corrections or flames from vendors whose products have been reviewed in this magazine. We reserve the right to not publish letters we deem in bad taste (bad language, obscenity, hate, lewd, violence, etc.).
Submit letters to the editor to Steve Litt's email address, and be sure the subject reads "Letter to the Editor". We regret that we cannot return your letter, so please make a copy of it for future reference.
 

How to Submit an Article

We anticipate two to five articles per issue, with issues coming out monthly. We look for articles that pertain to the Troubleshooting Process. This can be done as an essay, with humor, with a case study, or some other literary device. A Troubleshooting poem would be nice. Submissions may mention a specific product, but must be useful without the purchase of that product. Content must greatly overpower advertising. Submissions should be between 250 and 2000 words long.

All submissions become the property of the publisher (Steve Litt), unless other arrangements are previously made in writing. We do not currently pay for articles. Troubleshooters.Com reserves the right to edit any submission for clarity or brevity. Any published article will include a two sentence description of the author, a hypertext link to his or her email, and a phone number if desired. Upon request, we will include a hypertext link, at the end of the magazine issue, to the author's website, providing that website meets the Troubleshooters.Com criteria for links and that the author's website first links to Troubleshooters.Com.

Submissions should be emailed to Steve Litt's email address, with subject line Article Submission. The first paragraph of your message should read as follows (unless other arrangements are previously made in writing):

I (your name), am submitting this article for possible publication in Troubleshooters.Com. I understand that this submission becomes the property of the publisher, Steve Litt, whether or not it is published, and that Steve Litt reserves the right to edit my submission for clarity or brevity. I certify that I wrote this submission and no part of it is owned by, written by or copyrighted by others.
After that paragraph, write the title, text of the article, and a two sentence description of the author.
 
 

URLs Mentioned in this Issue