This is why off-site backups are a good idea

So Coding Horror and blogs.stackoverflow.com are experiencing "100% Data Loss"

Here is Jeff Atwood's twitter feed:

"looks like 100% data loss. thanks.. crystaltech.. :("
"you sort of assume your hosting company is competent. That's not a safe assumption in my experience."
"looks like it's 100% internet search caches for recovery. Any tips on recovering images, which typically aren't cached?"
"I had backups, mind you, but they were on the virtual machine itself :( I am OK on post-text, getting the post images is much harder.."

If it's on the same machine, it's not a backup. They should not have been on the VMs.

Oh, and my guess on the cause? SQL Injection

I like Jeff Atwood. I've been reading coding horror for years, and I've learned some great stuff. Unfortunatley, some readers may be feeling a little bit of schadenfreude right now, and here's why: There was a post on coding horror once that warned specifically against not having off-site backups.

It may not be his fault - His host may have told him there was a backup. However, it is a bit ironic.

What's Your Backup Strategy?

One thing's for sure: until you have a backup strategy of some kind, you're screwed, you just don't know it yet. If backing up your data sounds like a hassle, that's because it is. Shut up. I know things. You will listen to me. Do it anyway.

Update:

Looks like I was wrong on the SQL Injection cause!

ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images.

I wonder was else was on the physical VM server besides the SO blog and Coding Horror. Also, it's curious as to why the VMs wouldn't have failovers.

Update II:

Jeff has posted an awesome summary of events with some lessons learned. I'm meeting with the virtualization guys where I work on Thursday to discuss this (among other things)