Background Image

Blog Post

Courtesy of Wikipedia

Nov 17

How Sysadmins Can Prevent Dramatic Errors

We've all been responsible for something at some point. Website Admins, Linux System Admins, Bake sale operators? Whomever you are you have undoubtedly had at least one experience where something was organized neatly for a reason and had someone come along and screw it up royally by being ID 10 T. In new systems this isn't a massive problem, but on established systems someone with the wrong privilege can ruin quite a bit and quickly. Admins go to great pains to prevent this by believing in education until that fails - which it does in dramatic and costly ways.

The Typical Example  

New hire walks into HR and gets a badge and a password which grants user and some superuser privileges. Hotels for example might extend privileges to audit staff and for those hours in the night while sane people sleep the little underpaid auditor transfers numbers to a system on a shared server with countless other hotels in that franchise. One little auditor is more tired than usual and puts in the wrong ID for the location and suddenly a hotel in Quebec is credited twice, and one in Kentucky is not credited. Hours go by, the auditor is asleep but the error provokes a phone call to stunned manager who has receipts to prove the audit happened, and now needs a magnifying glass to find what went wrong. If this had been an IT job however - the consequences would be far worse, and harder to fix.

The Log  

Every user ID creates an entry and documents what was done and by whom. It sounds fool proof, but alas fools are invincible. Among other errors the wrong permissions can lead to some fascinating ways of covering up mistakes, like deleting logs. This may not happen daily, but towards the end of a bad enough day it can be a terrified employees last hope of hiding their mistakes. Because that's how some people think, and new hires are anxious little creatures, prone to attack like badgers when spooked by loud noises.

A Solution - Kind Of

No matter what kind of organization you are in charge of, buy a $20 set of walkie talkies for a mandatory probationary period of no less than 90 days. Watch the log intermittently, and stay in practically constant contact until you are 100% certain your hire doesn't freak out and sabotage to hide mistakes. Never react to mistakes the way you want to, instead treat errors as wonderful opportunities to calm down the whole team by having them spin tales of their worst moments and laugh about it. It's less expensive than watching the whole system go down when someone thinks unplugging the rack server might fix what they just broke. Go practice singing campfire songs through gritted teeth, it's less painful in the long run.

Add Comment:
Please login or register to add your comment or get notified when a comment is added.
1 person will be notified when a comment is added.