This is why off-site backups are a good idea

So Coding Horror and blogs.stackoverflow.com are experiencing "100% Data Loss"

Here is Jeff Atwood's twitter feed:

"looks like 100% data loss. thanks.. crystaltech.. :("
"you sort of assume your hosting company is competent. That's not a safe assumption in my experience."
"looks like it's 100% internet search caches for recovery. Any tips on recovering images, which typically aren't cached?"
"I had backups, mind you, but they were on the virtual machine itself :( I am OK on post-text, getting the post images is much harder.."

If it's on the same machine, it's not a backup. They should not have been on the VMs.

Oh, and my guess on the cause? SQL Injection

I like Jeff Atwood. I've been reading coding horror for years, and I've learned some great stuff. Unfortunatley, some readers may be feeling a little bit of schadenfreude right now, and here's why: There was a post on coding horror once that warned specifically against not having off-site backups.

It may not be his fault - His host may have told him there was a backup. However, it is a bit ironic.

What's Your Backup Strategy?

One thing's for sure: until you have a backup strategy of some kind, you're screwed, you just don't know it yet. If backing up your data sounds like a hassle, that's because it is. Shut up. I know things. You will listen to me. Do it anyway.

Update:

Looks like I was wrong on the SQL Injection cause!

ugh, server failure at CrystalTech. And apparently their normal backup process silently fails at backing up VM images.

I wonder was else was on the physical VM server besides the SO blog and Coding Horror. Also, it's curious as to why the VMs wouldn't have failovers.

Update II:

Jeff has posted an awesome summary of events with some lessons learned. I'm meeting with the virtualization guys where I work on Thursday to discuss this (among other things)

This is how to apologise for an outage

Remorse, Responsibility, Empathy, Recompense, and a Plan of Action:

Gmail's web interface had a widespread outage earlier today, lasting about 100 minutes. We know how many people rely on Gmail for personal and professional communications, and we take it very seriously when there's a problem with the service. Thus, right up front, I'd like to apologize to all of you — today's outage was a Big Deal, and we're treating it as such. We've already thoroughly investigated what happened, and we're currently compiling a list of things we intend to fix or improve as a result of the investigation...

[Read the rest at the Gmail blog]

Web Browser Standards Compliance Testing

As anyone that's done web programming can tell you, browser compatibility can be a headache. When I first threw html on the web 10 years ago, trying to get pages to render the same way in IE and Netscape was sometimes a challenge because W3C standards weren't followed consistently. Even now, compliance is an issue. Firefox, IE, Chrome, and Safari all take liberties with some standards, and the result can be a nightmare for developers.

Best practices dictate development in compliance with standards, and then managing any big problems with compatibility if required. Practically, some developers change style sheets for each browser or simply design their sites to look good in the browser most of their readers use and make tweaks when required for other browsers.


Acid2 is a test page developed to highlight the inconsistent adherence to W3C standards:

Acid2 is a test page published and promoted by the Web Standards Project to expose web page rendering flaws in web browsers and other applications that render HTML. It was developed in the spirit of Acid1, a relatively narrow test of compliance with the Cascading Style Sheets 1.0 (CSS1) standard, and was released on April 13, 2005. Like Acid1, an application passes the test if the way it displays the test page matches a reference image.

Acid2 tests aspects of HTML markup, CSS 2.1 styling, PNG images, and data URIs. The Acid2 test page will be displayed correctly in any application that follows the World Wide Web Consortium and Internet Engineering Task Force specifications for these technologies. These specifications are known as web standards because they describe how technologies used on the web are expected to function.

While at the time of Acid2's release no web browser passed the test, Acid2 was designed with Microsoft Internet Explorer particularly in mind. The creators of Acid2 were dismayed that Internet Explorer did not follow web standards and because of this Internet Explorer was prone to display web pages differently from other browsers. When such a discrepancy between browsers is encountered, web developers spend time tweaking their web pages in order to make the pages be displayed correctly in different browsers. Acid2 represented a challenge to Microsoft to bring Internet Explorer in line with web standards, making it easier to design web pages that work as intended in any web browser.


Check out the wikipedia article on Acid2 for some pictures of how the most popular browsers don't adhere to standards, and check out the site itself to see how the browser you're using measures up.

Updated: Firefox Beta Scores 93 On Acid3 Test (via Slashdot)

Decision Trees and Best Practices: COTS vs. Custom (Build vs. Buy)

Here are some points to consider when deciding between a commerical, off the shelf (COTS) system and custom development:

Is a COTS system available that meets most of the requirements?
(If yes, a COTS system may be a good option: (Avoid reinventing the wheel))

If so, is there a fully exposed API available?
(This is essential to integration / customization)

If so, is the source code available?
(This is essential to deep integration / customization)

Is the system designed to meet a core business function / create a competitive advantage ?
(If so, custom development may be a good option: See Joel Sposky's: In Defense of Not-Invented-Here Syndrome)

If so, will custom development allow for code reuse in future / other systems?
(There are many advantages to reusing existing code)

What is the TCO for the custom application vs. the COTS product?

Are there time constraints that could not be met with custom development?
(If yes, a COTS system may be a good option)

Decision Trees and Best Practices: Local vs. Centralized Processing (Thick client vs. Thin client)

Here are some points to consider when deciding between Local and Centralized processing:

Is a reliable network available everywhere the application will be installed?
(If no, local processing may be a better option)

Does the application require real-time data?
(If yes, centralized processing may be a better option)

Does the application require intensive CPU effort or high disk IO?
(If yes, centralized processing or a thick local processing may be options)

Does the application need offline capabilities?
(If yes, local processing may be a better option)

Are there security concerns for having the data consolidated in remote environments?
(If yes, centralized processing with distributed thin clients may be a better option)

Is there existing hardware at the sites that can be leveraged to run the application?
(If yes, centralized processing may be a good option: A thin client will cost less to deploy and maintain, they are more secure, and less valuable to thieves)

Are there existing applications that the same user group is working with, and how are these applications deployed?
(There may be economies of scale by consolidating applications into one design pattern)

Decision Trees and Best Practices: Web vs. Desktop Development (Online vs. Smart-Client)

Here are some thoughts on helping to decide between Online and Desktop developement. Although the line is being blurred by technologies like AJAX, Sliverlight, and Flex, there are still some times when one is better than the other

Criteria:

There 4 main factors to consider: Availability, responsiveness, clarity, and utility
Is the application targeted to users with different OS configurations?
(If yes, a desktop client may be a better option)

Will a web application be able to provide the availability, responsiveness, clarity, and utility that can be delivered with a desktop application?
(If yes, a web client may be a better option)

Does the application require OS-type functionality for sorting and editing data? (Although AJAX / asynchronous JS has made possible much richer desktop-like functionality to web applications, the desktop still offers advantages in this area)
(If yes, a desktop client may be a better option)

What is the primary objective of the app - data manipulation, or data viewing?
(If data manipulation, a desktop client may be a better option, if viewing, proceed to the next question)

What is the number of data points/records that need to be viewed/edited together on one screen?
(If the number is high, a desktop client may be a better option)

Does the application need offline capabilities?
(If yes, a desktop client may be a better option)