This story describes the aftermath of my leaving an untenable position
I quit my job, flipped my LinkedIn status to 'willing to hear from recruiters' flew home and spent a few days catching up with house/motorcycle/dental/car repairs, cat feeding and sleeping on a sopophoric couch.
I log back into LinkedIn to nothing I can take- temp to perm jobs three states away, sales positions and ill-formed "do everything like a senior but pay you like an intern" positions.
I even think about picking up some litigation work, but that smells like failure.
I drop a line to two technical recruiters I like and ask for some contract work to tide me over.
One, who we'll call Burt, calls me back that afternoon.
Burt:"Hey, I saw you were looking for short term work. I have an, er, odd request and you might be interested"
I hear a buzzing sound in the background, but assume it's just noise on the line.
me:"Ok, what is it?"
Burt:"This isn't one of my normal clients. They need to figure out their backups and possibly recover some data. It's urgent, so the faster you can get here, the faster you'll get paid. I'll guarantee that part if you make them happy"
I get a contact, an address and a phone number. I tell Burt I'll be there this afternoon and make my self presentable, collect some tools, then ride out there on my motorcycle.
I arrive, find a place to park and busy my self with getting my suit jacket and laptop bag off the bike while securing my motorcycle gear in my saddlebags.
A middle aged woman has left her beige sedan and is walking towards me with a "I want to talk to your manager" air.
Beige Camry:"Don't you know you'll never amount to anything riding that death machine?"
I'm still fumbling with my remove sunglasses, pull regular glasses from their case, place sunglasses in case, remove helmet, put glasses on my face without dropping anything or poking myself in the eye with the glasses, so I don't pay Beige Camry much attention.
Beige Camry:"You won't go to college and you won't make it past 30 doing stupid things"
I'm now paying attention. I'm still confused. I look around to make sure she's addressing me. She is. She's planted her nurse shoe wearing feet and wants to give me both barrels.
me:"I've made it past 30 and graduated college, but I still do stupid things. Thank you for the compliment"
I go back to the stowing and bungee cording while she glares, then walks off, towards a building next to the parking lot.
A few minutes later, I'm respectably wearing a suit jacket with my laptop bag slung over my shoulder looking for my destination.
I find my destination and the appropriate office suite. It turns out to be the Law Offices of Amygdala and Euripides, PC . The receptionist tells me to wait in a waiting room with outdated magazines while I hear the buzz of drills.
A middle aged man introduces himself as Mr Euripides and ushers me back to his cramped office.
Me:"So, I understand you have some problems with backups"
Euripides:"We do. We switched providers and now everything is messed up."
Me:"Ok. By 'service', we're talking a managed service provider- a contractor who handles technical issues, sets up systems and the like'?
Euripides:"Yes. The old one was too expensive. They were charging us rent on a file server, so we terminated the contract. I had my nephew build a new server and connect it to a cheaper backup service"
Me:"So, what happened?"
Euripides:" About two weeks ago, we lost all our files on the server and neither of the backups work. Each blames the other and they start talking technobabble. I threatened them with litigation, but that didn't get my files back."
Me:"I think I understand. I speak fluent technobabble. Let me take a look and I'll give you an estimate on the costs"
Euripides:"Burt is taking care of that part"
I'll have to figure out what that means. I'm here, so I should at least take a look. Lawyer hands me a wrinkled sheet of paper and walks me to a louder room. It seems that their IT closet is in a construction zone. There's sheet rock dust, saw dust and a few broken drop-ceiling tiles' worth of detritus on a single open rack.
At the bottom of the rack are two flashy gaming PC towers. Translucent Lucite sides with LED light strips show me a stack of hard drives . Clearly the builder was thinking about a different kind of Enterprise than I was.
I turn to Euripides.
me:"Ok. Any ideas on what each of these do?"
Lawyer:"My nephew's number is on that paper. I don't understand all this stuff anyway. Anything else you need from me?"
me:"Let me take a look and I'll call your nephew if I have any questions"
He walks off while I grab a spare keyboard and monitor from an unused cubicle. I plug into the PC on the left and use a password on the sheet to login.
A bit of poking and the OS sees one mountable volume, but four drives. Seems they're some kind of RAID. One drive is showing a hardware failure.
That's odd, but perhaps that's because Windows isn't my strength. I reboot the first PC from a linux USB drive and move my monitor and keyboard to number 2.
This one is a bit happier. I see two volumes mounted. One for the OS, one for file storage.
Even better. The 'storage' volume has one folder, labeled 'Backups'.
This might be easier than I thought.
I look through the backups and see one per day, going back about a month or two.
I pick one from about two weeks ago. It's compressed, so it's going to take a few minutes. I swap back to the first PC, which has booted linux.
Ok. Four drives. I can mount the OS drive fine. One dead. Remaining two a part of an array. I don't get it. Assuming RAID 5
, one drive can fail and we can still read and write data. I need to think on this for a second.
Back to the other PC to see how the restore's coming. That was fast. I open the folder and see a full backup.
Of the operating system. Not the file storage. I check the other backups and they're all the same size. Great.
Then I notice something that makes me sad. The OS volume is 4TB. The Storage volume is 12TB. One drive, three drives.
There's no parity, the drives aren't mirrored. If any one fails, the whole volume fails.
I find Lawyer's nephew's number and call it. I get voicemail. I tell him to call me back.
So my local backups aren't and the one volume that holds the data is borked.
This is going to be complicated. I call Bart, the recruiter who brought me in.
Bart:"Hello! How's it going, LT?"
me:"Uh, not so well. I have a feeling this is going to be expensive."
me:"We might have to ship out three drives for recovery. We're talking a few thousand to start"
Bart:"Is that what you'd charge?"
me:"I don't have a clean room. I might swap a hard drive controller board, but this is magic by comparison."
Bart:"Is there anything you can do?"
me:"I have one last thing to try, but I'm not feeling lucky. Give me an hour"
Bart:"Let me know"
me:"Oh. One last question. Are you paying me?"
Bart:"Yep. I'll pay you for your time and barter with Euripides"
me:"I won't ask"
I look at the other number on the paper. It's a local MSP named OnFight. I call the number. After a few minutes in a phone tree inspired by the maze in Zork 1, I exit the twisty passages and get to Joel, a support rep.
hold music, occasionally interrupted by short, amateur commercials about how awesome
OnFight's service is.
me:"I'm trying to reach whomever manages your backups. I don't have a customer number, but the customer is Law Offices of Amygdala and Euripides, PC."
me:"Wait wait wait. Can I talk to a manager for five minutes? I think can solve a problem for you all if I can talk to someone who can make a quick decision"
The wait isn't as long.
Manager:"Hi there. This is Rebecca. What's this about Amygdala and Euripides?"
me:"Thanks. You used to rent them a server and do their backups, right?"
Rebecca:"That's correct. A few months ago we got told to take the server back and to stop the backups"
me:"Ok, got that. Is there any chance you haven't wiped that server?"
Rebecca:"We overwrote the drive, reformatted the server and allocated it to a new customer. Backups securely overwritten per A&E's request"
you to delete everything?"
Rebecca:"That's correct. We have emails from Euripides' nephew and a signed letter from Euripides himself"
me:"Wonderful. Thanks. I'll make sure they understand"
I call Nephew again.
me:"Hi. This is Lawtechie. I'm trying to figure out your backups"
Nephew:"Yeah, this all should work"
me:"Let me understand. The shiny box on the left is the primary and the one on the right is the backup?"
Nephew:"Yep. What's wrong with you that you can't get it back?"
me:"It looks like you're only backing up the C: drive"
Nephew:"What do you mean?"
me:"You're backing up the operating system on a nightly basis"
Nephew:"Right. The important stuff"
me:"No, that's reinstallable. What about the various documents your uncle's law practice actually runs on?"
me:"Unless there's another backup server, no"
Nephew:"What do you mean?"
me:"There's a lesson every techie learns. If you don't test your backups, you don't have any"
Nephew:"We shouldn't need them. Those disks are RAID"
me:"You sure? It looks like you set them up without mirrors or parity"
me:"I think you should learn some things before building systems that people rely on next time"
Nephew:"Yeah. I'm going into cybersecurity soon so that stuff won't matter"
me:"Good for you. That's a business where charlatans and the soulless run rampant. There's also a negative side."
me:"Is there any chance there's a full backup anywhere else somewhere?"
Nephew:"Yeah. OnFight should have one"
me:"Tried that. Seems that you told them to burn it all"
Nephew:"It's more secure that way"
me:"I wish you luck in your career"
Nephew:"Don't throw me under the bus"
I texted Bart to tell him that it wasn't looking good, then I walked to Euripides' office to tell him about hard drive recovery. He thanked me in the way people do when they think you're trying to take advantage of them and said that he'd talk to his nephew before making any important decisions.
Bart paid me for that day and I went home.