Setting the story straight

On March 21, The Boston Globe reported that Carbonite is suing Promise Technologies, a company that makes storage servers that we purchased back in 2007. This lawsuit stems from an incident that occurred over a year ago. The article (and subsequent coverage by other outlets) references court documents which say that Carbonite "lost the backups of over 7,500 customers." It is possible that readers will walk away from this with the impression that 7,500 customers were unable to restore their files from Carbonite. This is not the case. Let me explain.

All of the affected customers had their backups re-started immediately and automatically. Statistically, about 2 out of every 1000 hard drives will crash every week (about the time it took to get most customers backed up again), so a small number of these customers had their PCs crash before their re-started backups were complete. These customers were unable to restore all of their files from Carbonite. We took full responsibility for what happened, and I did my best to apologize personally to each of these customers.

For the techies who are reading this, what happened is this: The Promise servers use a technology called "RAID" that spreads data redundantly across 15 disk drives so that if any one disk drive fails, you don't lose any data. In fact, the kind of RAID we use allows us to lose 3 of the 15 drives simultaneously before you lose any data. This configuration is in theory 36 million times more reliable than a single disk drive — the chances of 3 out of 15 drives failing at the same time are almost nil. The RAID software that makes all this work is embedded as "firmware" in the storage servers that we buy. In this case, the firmware had bugs that caused the whole server to crash.

So that, in a nutshell, is what we allege in our lawsuit. We were sold defective equipment and hence have asked Promise to refund our money. So far they have refused to accept responsibility, so now we are suing them. The Dell RAID servers that we started purchasing a couple of years ago have been flawless and we're extremely happy with them.

Dave
CEO, Carbonite

Related posts

Comments

March 23. 2009 18:31

Naveen K

This explanation is still not reassuring. When I see the word failure next to "36 million times more reliable", I begin to worry about remote backup. Why not back it up to 2 different locations, that don't communicate, so you don't have the domino effect?

Also, you have to stop using "this was when we first started" as excuse every time press has a story showing Carbonite in bad light.

Naveen K

March 26. 2009 15:06

David Friend

Naveen:

There’s no limit to how much money you can spend making backups of backups. Backing up the backup doubles the storage costs. We have never seen three drives in an array plus the user’s PC fail all at the same time, and we are Dell’s biggest customer worldwide for their RAID storage array. I doubt that we ever will. BTW, the chance of an external hard drive failing is about 10% per year.

Having copies both in our data center and on the user’s PC solves the “two cities” problem -- if a A-bomb takes out our data center, you still have your PC. If it takes out your PC, then we'll still have it but, well, you probably won’t need the backup.

Here’s what you should be worried about: software and firmware bugs and human error. You know about the firmware bug – that’s why we’re suing our vendor. But the biggest problem BY FAR in our market is human error – somebody forgets to include a particular folder in their backup set, and when their PC crashes they discover that the files never got backed up in the first place. It may be easy for skilled people like you to create backup sets, but in our market of consumers and very small businesses, 40% of our users have never even used their right mouse button! So we solve that problem by simply backing up everything for a flat fee. This has hundreds of times more impact on the customer likelihood of a successsful restore than making what is already a very remote possibility even more remote. For those people who feel they need backups of backups, there are plenty of services out there. But they will cost for one month what Carbonite charges for a year.

As for the timing issue, I'm wish I didn't have to correct sloppy reporting. When TechCruch says "Online Backup Company Carbonite loses Customers' data" that sounds like it just happened and is a current problem. Same with the number of customers who lost data: 54, not 7500. Their readers certainly thought so, based on our email. And it does matter that these things happened years ago and the problems have been resolved. Every company has screw-ups. What matters is how you deal with them and whether you fix the problems and make good for anyone who was adversely affected.

It is very dispiriting that out of the hundreds of blogs that jumped this story, NOT ONE bothered to pick up the phone to ask me if the facts were true. My contact info is right on the Carbonite web site.

David Friend, CEO
Carbonite, Inc.

David Friend

May 3. 2009 01:40

Lee

Hi David,

I'm probably going to subscribe to Carbonite in the coming weeks. I understand most of what you are saying here (and agree with your statement that doubling backup doubles costs), but I have a question about the above topic. You state:

"We have never seen three drives in an array plus the user’s PC fail all at the same time, and we are Dell’s biggest customer worldwide for their RAID storage array. I doubt that we ever will."

But isn't that what happened to those 54 customers you mentioned? Or are you making a distinction between the drives failing and the firmware bugs. If you are, that's kind of a meaningless distinction, isn't it? All that mattered for those 54 customers was that your backup service failed and they could not retrieve their files, right?

Lee

May 6. 2009 18:20

David Friend

Lee,

As far as the 54 customers are concerned, you're right: all that matters to them is that they couldn't get their data back. But the distinction IS important. If you are losing data because of disk failures, it says that your redundancy strategy is inadequate and it will surely happen again.

We picked RAID6 because it is one of the most reliable disk storage strategies on the market. It is the backbone storage architecture for big data centers the world over. It is so reliable that statistically there is less than a one in a thousand chance of losing data to disk failure in your lifetime. Meanwhile, your PC will fail on average once every 3 years. And an external hard drive has about an 8% chance of failing just in the first year and external drives frequently succumb to data-destroying viruses (to which services like Carbonite are immune).

David Friend

Comments are closed