Backups suck (but we need them)

Backups are hard to do, boring, thankless tasks, and often one of the things that gets pushed to the back of the pile. And yet protecting data is probably the most important responsibility for most SysAdmins.

The computer systems we install, maintain, and our users rely upon daily continue to store more and more data. Disasters and accidents will happen, and will lead to you losing some or all of that important data.

I have dealt with some unusual and unexpected data disasters – burst fire sprinklers, computers falling from a plane cargo hold, disappearing online storage providers and a few exploding CD’s. There have also been the more mundane drive failures, viruses, accidental overwrites, and the ever annoying “My spreadsheet disappeared”. So far I haven’t had to deal with a total site loss or any other major damage, but you need to be prepared.

But I do backup!

So, you have regular, automatic and regularly tested backups. That is awesome, and you rule! Unfortunately there is one area that often gets overlooked – backing up new, modified, or moved services.

We both know how it happens. You are in a rush to get things done, your boss want this service running yesterday, and you sort of overlook backing it up. After all, you will do it later, right? Other projects and distractions get in the way, and automated backups often don’t get implemented until after the annual DR audit (if you don’t have one, establish one).

In the meantime the hard work you put in installing, configuring and maintaining the new service is at risk. Oh, and that pesky user data too. You might not think backing it up is important, but it really is. It just takes one bad SQL query, a single disk failure, maybe a simple virus and it’s all gone. And if there is no data, then the company really doesn’t need you any longer.

Plan to backup (and restore)

So, how do we prevent these sorts of mistakes? The biggest thing we can do it plan to backup. “What are we going to backup, and how?” should always be two of the first questions asked when first thinking about adding/moving or modifying services. Make backups part of your standard project checklist (don’t have one? Make one).

If you have a major new service being rolled out, or a significant change then implement (and automate) backups as you complete each major milestone. You would not be happy losing even a weeks worth of work because backups aren’t implemented yet, and losing data could be far worse.

Test that restores work as you implement each part of the new backup. Otherwise you won’t know if it works! Also document the project and how you back it up while your memory is still fresh, in 6 months you will be stoked you did.

The Good News

The good news is planning to backup as a part of your project has some very real benefits:

• Minimize risk – Reduce loss of data, morale, face, time and money.
• Easier budgeting – Backup becomes part of the project and not an unexpected cost.
• Easier documentation – Having a plan makes it easier to document the how and why.
• Keep your boss happy – She hates having to explain problems to her boss.
• Prevent the ‘oh-crap’ moment – Sysadmins have enough stress to deal with.
• Better planning – Planning your backups will often help find other things you didn’t think of.

The very best part is, planning and implementing your backups from the start makes them easier to deal with in the future and less stressful. Who knows, you might even enjoy it.

Links

ServerFault blog – putting a value on backups

Mr. Backup Blog

Everything Sysadmin – The Limoncelli test: Backups

Amazon.com – Backups & Recovery by W. Curtis Preston

Usenix – Backups and Recovery by W. Curtis Preston and Hal Skelly