Here's some moderately big news in the online backup industry: HP has decided to discontinue their HP Upline service. HP got into this business by buying a company called Opelin. They spent about a year fixing the product up before launching it as HP Upline. I tried the service when it first came out and found it to be a little complicated for the average consumer. Only a few days later, they experienced an embarrassing outage where the whole service went down for several days. This was followed by a string of other software failures.

I'm not sure what HP was thinking when they decided to get into this business in the first place. They aren't in the antivirus business. They're not in the operating system or firewall business. What made them think that they could build and operate a world-class online backup service?

Everyone thinks that building an online backup service is easy. When I was raising money for Carbonite, one venture capitalist waved me off saying, "Any engineering team could design a product like that in a couple of weeks." When I was teaching at MIT seven or eight years ago, one of my students was fuming about Google's success, saying "Anyone can write a search engine. What's the big deal with these guys?" Maybe so, but writing a search engine that can process tens of thousands of searches per second against a database that is bigger than all the world's libraries combined is not so easy.

Same thing holds for online backup. We have 60 man years of development in the platform that stores your data. Data comes pouring into our data centers at a rate of about 100 million files every day, or 70,000 files every minute around the clock. The software and architecture that allows all that data to get onto our redundant storage arrays without losing even one byte is incredibly complicated. So while it's easy to back up one PC with one external hard drive, backing up many hundreds of thousands of PCs that are all simultaneously sending you data from every corner of the earth, is quite another matter.

I can understand why HP would want to put online backup on their PCs – when an HP PC fails and you lose all your wedding pictures, you are probably going to be mad at HP, not at the people who make the hard drive inside the PC. But I don't understand why HP wanted to own a service like this in the first place. They get their antivirus and other services from 3rd party vendors, and they should have done the same with online backup. I guess they learned it isn't as easy as it looks.

— Dave
CEO, Carbonite

I'm sure that many of you have read that HP’s online backup service went down shortly after it was introduced. It’s back up now, and HP has explained that what they experienced was a "technical glitch." Here are a few links about this story:

Enterprise Storage Forum: HP Upline Suffers Downtime

InformationWeek: HP Shuts Down Online Storage Service

Beta News: Bringing down the cloud: HP's Upline down for a third of its life

Why am I blogging about HP’s problems? Only because it underscores the difficulty of building a reliable large scale backup service.

When I was out raising money for Carbonite, one venture capitalist dismissed backup as a "trivial" application. It reminded me of an incident when I was teaching at MIT a few years ago: one of my students insisted that Google wasn't worth billions of dollars because "anyone can write a search engine." It’s true. But Google's barrier to entry is their huge scale. To build a backup service that can flawlessly store and retrieve billions of files is not so easy, as HP has learned.

When we first started out, we were using Microsoft’s NTFS file system. When we got to about 500 million files, it started to crash and gave us all kinds of problems. When we called Microsoft for help, the engineer on the other end asked us how many files we were storing. When we said "about 500 million", there was silence on the other end of the phone. He said "Uh, well we didn’t really design NTFS for that many files." So we set about building our own proprietary file system that could handle more than a trillion files, because that’s where we’ll be in a couple of years. We currently receive almost 40 million new files every day. And we have close to 7 billion files backed up. We restore millions of files every day.

To do all that without losing any data is an enormously complicated engineering challenge. We're three and a half years into this effort and there is no shortcut way to get to where we are, no matter how much money you throw at the problem. Our confidence in our own infrastructure is due to the fact that we’ve built a customer base of hundreds of thousands of users. Until a company does that, they’ll never know whether their systems are going to fall over. HP, with all their resources, is going through the same learning curve that we’ve gone through for the last 3 years.

— Dave
CEO, Carbonite