[ Site Index ] [ Rant Index ] [ Feedback ]
A Backup Crisis
It's a quiet afternoon in January. I'm alone in the support office, playing with my new yo-yo. The new DAT drive is whirring happily in the background, as I'm doing a fresh level 0 backup of the web server -- not our customer's files, heaven forbid, they might ask us to restore something, but the OS and all the neat-o software I've installed on it (like Doom).
The phone rings. I sigh and pick it up. "Hello, Acme Web Services here, HowCanIHelpYou?"
The voice at the other end of the phone is easily characterised: luser, male, type 3 ("I don't understand this shit but I'm not going to lose face, so I'll bluff"), probably a marketing executive.
"Hello, X here. I've had an account with you people since last month. It's about my files. The links don't work."
"Oh?" I ask, intrigued. "Why don't they work?"
"You're so bright, you tell me," he says.
"Certainly, sir."
I fire up emacs and poke around his home. Yep, there's a public_html directory in it. I go inside. Yep, there are seven files. file(1) tells me that two of them are GIFs -- funny, their names end in .jpg. The other five all have interesting names, uppercased, with eight leading characters then a .HTM suffix.
I look at my desk diary. Yep, today's least favourite OS: you guessed it!
"I'm having a look at your files now sir ..."
I open up all the html files. They're full of links. Unfortunately he doesn't seem to be clear on the concept of closing his anchor tags, so I tidy things up by deleting them all until sgmls agrees that the files are good HTML. (Of course, he wrote them using Netscape extensions so there's not a lot of valid HTML 2.0 left after I get through with them, but what the hell ... at least his link to Lycos still works.)
"That should work. Now all you have to do is change all your file names so that they've got the right suffix and you should be okay."
"What do you mean?"
"It's like this: when one of your lusers requests a file from our server, our server looks at the filename's suffix and works out what type of file it is. It then tells the luser's browser what to expect. Unfortunately there's no such thing as a .HTM file -- you'll need to change them all to .html, which is the correct suffix for a web document. Got that?"
"Frankly, no. Why should I change my filenames? You can't give a file a four-character suffix, anyway. It's not legal. Why can't you fix your broken server?"
Oops. (Am I feeling generous today? Hmm ... let's find out.)
"Pardon, I don't think I heard that," I say.
"Why don't you fix your file server so it knows about HTM files?" he repeats. "After all, we Windows users are in the majority."
"Of course," I say, going icy cold. "Sure. I'll try and patch the MIME content-type handler. Of course I'll need to use a binary editor and it may take a while, but rest assured that I'll see that your .HTM files get the right MIME type without you having to do a thing."
"Thank you!" he says, and rings off.
I turn to the keyboard. Clickety-click ...
perl -ne "s#HTM\ttext/html#HTM\tapplication/gif# && print;" \ <$HTTP_BASE/mime.conf >$HTTP_BASE/mime.conf kill -1 $(ps -gaux | grep -v grep | awk -e '/httpd/ { print $9 }')
... that should see to his content-type problems.
I go back to playing with my yo-yo. It's one of those transparent acrylic ones with a chamber inside for some LEDs. I've sort of modified it to conform to the original design specs for the yo-yo, which was -- as any fool knows -- invented as a weapons system; a kind of aboriginal LART. The mouse in the ballast chamber is looking a bit sea-sick by now; I think he's near to puking. Another few bounces and maybe we'll have some rodent vomit to experiment with ...
Then all the lights go out and I am surrounded by the hum of disk drives spinning down.
I'm out of my chair before the rodent hits the floor. "WHO CUT THE FUCKING POWER OFF???!?!!!???" I scream at the top of my lungs. (We're a shoe-string operation, and the Boss from Hell -- bless his pungent little sense of humour -- thinks that an uninterruptable pun supply makes for a good power backup policy.)
"Er, anyone up there?" someone calls from downstairs.
I wander out onto the landing. Being the webmaster in this joint, I get to starve in the dusty garret two stories above the neat front offices with the cute receptionists and rubber plants. Today's a Saturday ... nobody else is home.
"Yes! Who the hell are YOU?" I call.
"Electricity company. Er, someone asked us to install a new distribution board down here?"
I'm down the staircase so fast I leave scorch marks in the carpet. "Someone asked you to WHAT???!!!!????" I scream.
The electrician dude cowers. "I didn't know you were up there! Were you using any electric appliances?"
"Only the ENTIRE INTERNET FEED for OUTER MONGOLIA!!!???!!!!" I scream, and whack him upside the head with my yo-yo. The yoyo breaks open. The electrician falls over, gurgling incoherently. The mouse has finally had enough; the tiny sound of murine projectile vomiting tickles my ears.
Let's see. He's switched off the main breaker on the distribution board. That's going to take, oh, a good few kilowatts. Hmm.
I flick off the breaker on the clean loop that supplies the machines then switch on the main power supply.
CLICK. The lights go out again almost immediately. Fuck. I need power, and fast. The circuit breaker won't take it, and I don't have time to go round switching everything in the building off then power up in sequence. There's only one solution. I unplug the breaker and wire the electrician in series with the terminals. A bit of rodent vomit to wet the contacts, and I can throw the switch. He jerks around and smokes a bit, but I hear a heart-warming chorus of beeps from upstairs as the network begins to come back up.
Back in the machine room I prepare to deal with the impending crisis. First I take the phone off the hook. Then I check the servers. All the PCs seem to be okay, but the ancient 386 (running NetBSD) that we keep the web server on appears to be a bit sick. This could be bad news. I wait for the Linux boxes to come up, then amble over to the web server and look at it.
No, it's not power trouble. The fucking mouse pissed on the keyboard, didn't he? I knew I shouldn't have fed him those tequilla slammers.
Well, the BSD box is well and truly wedged as long as there's a keyboard on it, so I pull out my trusty leatherman and cut the keyboard cable, then hit the reset button. Aha! Signs of life ...
It boots. Goes multiuser. Fires up apache. And begins playing NIHIL by KMFDM at full volume. (I _knew_ the Boss from Hell had hidden that multimedia kit somewhere obvious!)
I put the phone back on the hook. It begins ringing, with call waiting, so I let it tinkle for a bit until the answerphone kicks in to take caller #1; then I pick up caller #2.
It's my luser du jour again, and he's pissed. "My links don't show up at all, now!" he yells in my ear. "And all my beautiful BLINK tags have gone! How can I highlight the IMPORTANT SPECIAL FREE GIFT OFFERS on my site if my customers can't see my BLINK tags? This is costing me a thousand dollars an hour while it isn't working, and I'm paying you good money -- all of ten bucks a month -- for this service! Fix it! Or else!"
"Excuse me," I say, "but we're in the process of recovering from a major power outage caused by the local electricity company which led to a localized blackout in the past half hour. There may be some transient damage to files on the server -- bit rot, you know -- and you really ought to upload your backup copies of the files if you want to be sure that they work."
"What local backup copies?" he foams.
"The copies you wrote on your machine and uploaded to our server," I explain patiently.
"But I erased them to reclaim space after I uploaded them to you! Can't you get them back from some backup?"
I glance at the DAT drive. Eight gigabytes ... "I'm afraid not," I say. "Our customer's files are their personal property and therefore entirely their responsibility. We don't look at them unless we're invited to, and it would be gross misconduct for us to make illegal copyright-violating backups of them. Why, if we backed our files up you could sue us! So we don't do that."
"AAUURGH ... WhatAmIGoingToDo?"
INTERLUDE
Now, Gentle Reader, it is time to take stock.
I am, as you know, the Psycho Webmaster From Hell. But I am not a ravening serial killer. I may have slain one luser today, but I think you'll agree he deserved it in spades. This guy, although he's a card-carrying nuisance, is paying my Boss money which eventually goes into my pay packet. It is therefore Bad Practice to kill him, because he'll stop giving us his money if I do that. Therefore, I need to find a way of shutting him up while simultaneously keeping him as a customer, convincing him that we offer a first-class service, and punishing him for his temerity in disturbing my Saturday-afternoon nap.
The solution should be obvious ...
"Excuse me, but are you using Netscape?" I ask him.
"Y-yes. Why?"
"Well, it occurs to me that you probably have backup copies of your web files knocking around," I say. "Have you previewed the files in Netscape? All of them, recently?"
"Yes ..."
"How big is your Netscape cache?"
"I keep it set to use all my free hard disk space," he admits. "That's why I had to delete the files to make room."
"Ah, capital! Then you've got ... how big a disk?"
"Two gigabytes."
"Ah, so your Netscape cache is less than two gigabytes in size. Excellent! Listen, whenever you look at a file, Netscape keeps a copy of it in your cache directly. The files are all numbered and there's no easy way of figuring out what they are by name short of looking at every one of them -- but you've only got two gigabytes of search through. Unfortunately you can't use Netscape to look inside it's own cache directory. But that needn't stop you; all you have to do is open each of the files in turn using Windows Notepad, then read them to check whether they're the files you wrote; if they are, copy them to a floppy and give them back their original names."
"Oh, you're so wonderful!" he sobs. "I just might keep my job because of you! I'll remember this!"
He rings off, a happy hunter. I briefly ponder whether I should have introduced him to the mysteries of grep(1), but conclude that I did the right thing.
After all, a little knowledge is a dangerous thing, especially in the hands of an incorrigible idiot.
[ Site Index ] [ Rant Index ] [ Feedback ]