This blog is about three months in the making.

First, a bit of background. Several posts ago, I predicted the death of SATA in favor of SAS, which is only marginally more expensive (not talking the dirt-cheap integrated SATA controllers, but higher-end cache-carrying SATA RAID controllers) for an admittedly smaller capacity but much higher speed.

After using SAS on some of the servers and blades at work, I came home to my SATA-based desktop computer and wept silently whenever I did anything disk-intensive, because it was soooooo much slower. I have SCSI for the OS in all my server equipment, but even those machines weren’t as peppy as the SAS stuff at work. Taking these two things into account, plus the fact that the games I like to play are all disk I/O intensive, then throwing in a bit of friendly rivalry for good measure, I decided to upgrade my desktop machine to use SAS storage.

Read more »

I feel the need to make a confession here. Up until yesterday, despite spending a generous portion of my waking hours covering data backup, disaster recovery and data protection, I myself did not have a backup plan.

I do digital photography in my spare time, and creative writing outside work, and I’ve been a digital music addict since the advent of Napster. So I have about 100 GB on two IDE drives inside a Windows XP machine custom-built for me by a highly geeky friend. And it’s just been sitting there, waiting to be snatched away into the ether.

Then another friend of mine told me about how his MacBook hard drive crashed. On his birthday. While he also had the flu.

He told me how his entire visual design portfolio, an important part of his resume for the business he’s in, has been lost, along with all of his digital photographs, many of which he didn’t have posted on Flickr or stored anywhere else.

He went on to tell me that his costs for trying to recover the data from the drive are going to run him upwards of $2,000–if he’s lucky. It could be cheaper, but that would mean less of his data has been recovered, and so now he finds himself in the position of hoping he’ll have to spend more money.

It’s a bittersweet subject for him that so many people he knows, myself included, have credited his experience with finally getting them off their butts and backing up. But that’s the reality.

I ended up going with the 500 GB Western Digital MyBook, because that’s what my friend also ordered once he learned his lesson the hard way, and he’s far more technical than me, so I trust his judgment. The MyBook came with Memeo’s AutoBackup and AutoSync software, of which I’m only using the former. It also came with a bunch of Google software including Google Desktop, which I found rather odd.

Having covered data storage for the enterprise, I’ve had a chuckle whenever I’ve checked on the initial backup job’s progress. Granted, it’s got a QoS feature that cedes system resources to the PC, but let’s just say I’m not seeing the kind of data transfer rates with this thing I’m used to hearing about. It’s been funny, after being immersed in systems that perform at 8 Gbit or 10 Gbit for a few years, to watch my little PC poke along at what seems like 1 MB/hr, if that.

But still. At least I have a backup. Finally. And I can finally rid my closet of that skeleton.

Now my issue becomes off-site disaster recovery. It’s far more likely that my hard drive(s) will crash than that my house will be napalmed or something (knock on wood), but no sooner had I told Tory that he could stop bugging me about backup, than he started bugging me about taking the drive to my office once the data transfer is done.

But the AutoBackup software, like so many low-end and consumer backup offerings, is set to automatically backup changed files, and what I told Tory was, I like having a low RPO over here. And I made that napalm comment, I’ll admit (I can just feel karma coming to get me). So I’m thinking about some kind of backup SaaS for off-site DR, but capacity with those services is at a much higher premium than it is in 3.5 inch external SATA. And so you know what that means…data classification!

I may be poking along at 1 MB/hr, but it all feels like a slow-motion, small-scale version of the issues I cover every day. It’s interesting to see firsthand how ”Digital Life ™” is, in fact, blurring the boundaries between home and business computing.

Xiotech put out an intriguing press release yesterday, headlined “IT MANAGERS EXPRESS CONCERN ABOUT STORAGE SCALABILITY AND CAPACITY.” It discusses survey results from end users establishing that reliability is a key feature when considering storage systems (I personally prefer the ones that crash all the time, but maybe that’s just me), and an ominous quote from Xiotech’s CTO:

“The industry is on pace for a data storage crisis in the next few years,” predicts Steve Sicola, chief technology officer at Xiotech. “The cost of adding a gigabyte of storage is dropping nicely every year, but the cost of managing, protecting and servicing that storage continues to grow. Drives are the most numerous constituent of data centers, and with that have the largest probability of failure. If demand for storage continues at the expected pace and nothing is done, we may see a significant increase in data loss and accelerating cost inefficiencies.”

Thinking Xiotech may have joined Carnegie Mellon University and Google in blowing the whistle on drive reliability specs produced by manufacturers, I hopped on the phone with Sicola this morning. The call began with compelling candor. “Drive makers and systems vendors may say all drives are standard, but they don’t all talk the same way,” Sicola said. “People don’t see the problems that are already happening because big systems houses are trying to make money on both the front-end and the back-end.”

So are we saying that big systems vendors are covering for drive manufacturers even more than the Carnegie-Mellon or Google reports already led us to believe? Was there a heretofore undiscovered problem with drives Xiotech wants to warn us about? “It’s not one specific problem with drives, it’s when you add up a lot of drives in one system that you have a higher potential for failures, and more time spent addressing drive failures,” Sicola clarified. Ah. Hasn’t storage growth been raising the potential for failures over the last couple of years? Isn’t this why we have RAID 6 and clustered systems and. . . .
That’s when we got to the heart of the matter. Xiotech is also coming out with an approach to addressing storage growth, built on IP acquired from Seagate’s Advanced Storage Architecture (ASA) group in November. This itself isn’t news, either. We covered that acquisition when it happened.

Now thinking that the press release was a lead-in to a deeper discussion about what ASA will do and how Xiotech is developing it, I began asking questions along those lines. To my surprise, at that point, you’d have thought I had initiated this call, with the goal of getting Xiotech to divulge trade secrets. No comment, no comment, no comment. No comment on what specifically ASA will do to provide better storage reliability, no comment on when we’ll see ASA released by Xiotech, beyond “later this year.”

“So,” I asked, “until that happens, isn’t Xiotech one of those systems vendors making money on unreliable back-ends, too?”

“We see this problem really exploding in the next few years,” Sicola clarified again. “Xiotech has done a lot today to ensure balanced configurations and has limited the size of its systems. I’d be more concerned about big systems vendors like EMC and Hitachi that are packing in so many drives, which is like trying to herd 1000 cats.”

So to review, the news today seems to be that Xiotech is eventually coming to the rescue of PB-plus shops dealing with reliability issues. Sometime soon-ish. As to how they’ll rescue you. . .well, you’ll find out when they get here. Hopefully.

I was talking with a friend the other day about the prospect of multi-terabyte hard drives and how painful it would be to lose that much data. My friend — being my friend of course — countered that it’s not the amount of data, but where it resides and what the data is that’s important.

For instance, he went on, the EEPROM on your desktop motherboard isn’t more than 2MB worth of data. Yet without it the bazillion hours of work you have stored on your desktop hard drive, while safe and sound, is still useless to you because you can’t access it because your computer won’t boot.

After conceding the point, I rephrased the statement to emphasize the loss of multiple terabytes of data residing on a platter-based spinning medium, located in a computer or computer-like device providing data storage services to said computer, group of computers, or computer-like devices (whew!).

Without blinking an eye, he said he’d started a hard drive data recovery company. He built a clean room and had been perfecting his recovery skills on hard drives purchased on, all of places, eBay. As an aside, use a hammer and nail, or Sawzall, to properly delete all data from unwanted hard drives you dispose of.

A while back, I got a frantic call from a family member whose laptop hard drive had crashed. She was beside herself because on her hard drive were all the digital photos she’d ever taken. . .ALL of them. She’d meant to back up her stuff to a disk but never got around to it. She wanted to know was there anything I could do to help her.

That is when it hit me full force, I have brilliant and baleful friends.

My friend recovered almost all the data from her hard drive for me (at a very reasonable price) and now she has the first pictures of her child, some of her wedding photos and other very important moments in her life back, and on DVD this time. The whole saga got me thinking: Am I really protected from a hard drive crash? How about the executives I support? What would I do if my array at home failed where I have all of my photos!

Seeing the look on my relative’s face when I presented her with all of her photos was priceless. But it got me thinking about all the other people out there in the SMB world with the 0.5 person IT shop who don’t even know these services exist, much less who can afford the super-high cost of traditional data recovery. I don’t think today’s data protection schemes are going to be able to handle the eventuality of these super-sized drives making their way to the same SMB shops.

Do the math. A decent 100Mb pipe can push about 3TB an hour (this takes into account -25% for packet and transmission overhead). If you had three people with a terabyte drive, you’d saturate a 100MB uplink should they decide to back up to a device on the network. How are we going to back that up? The storage SaaS startups making their way to market aren’t going to be able to keep up either. Imagine backing up 400-700GB over your home Internet link where your upstream bandwidth is only 768Kbps.

I saw this coming a bit back when I got my grubby hands on the Hitachi Terabyte drive and have begun using a combination of VMware Player and VMware Workstation to mitigate my issues with capacious storage at home. I essentially virtualize the machine I want to use and deploy that on top of a generic OS install, replete with a pretty icon (in my case, Debian Linux), instructing the user to launch the player as their “desktop.” I’ll eventually get to a point where I will move upward from Player to Workstation for all my machines (right now cost is limiting me to using player for most of my machines), then run snapshots and back up the snaps to the same location as the original VMDK using RSync.

It sounds like a lot of work, but try explaining to your wife that she’s lost all her projects she’s been working on and you don’t have a recent backup because her drive is too big to back up quickly. You’ll appreciate the effort that much more when you can say, “I’ve got you covered, hon!!”

Here’s the visual I use when I explain this concept.

1) Fold a piece of paper four times (or use a folded napkin)

1a) Imagine the paper (napkin) as your physical hard drive

2) Tear off two or three 1-inch pieces of that napkin. Put them on the table next to the napkin.

2a) Imagine those pieces as virtual hard drives or volumes.

3) Reorder those 1-inch pieces of the napkin. Easy, isn’t it?

4) Peel apart the layers of those 1-inch pieces, 4x as much stuff to manipulate, making it take a little longer to move things around the table, no?

4a) Imagine those layers as individual files.

Take this one step further. Blow a soft puff of air at the three 1-inch pieces before you peel them apart (this works best with the napkin as they are slightly “stuck” together). Think of that puff of air as a failure or some sort of issue with storage. Do the same when you’ve peeled apart the pieces.

Now you have a great way to envision how your task of managing individual files (family photos) on a gargantuan hard drive (look how much napkin you have left!!). Multiply that out by a couple of napkins and you see why all of a sudden this problem of failed drives and how to protect against it becomes really hard in the TB-drive world. This can open eyes at the management level. It puts a real and appropriate understanding of why we as storage admins freak out at times when they refuse to allocate budget.

I started out talking about the advent of huge drives and what are you going to do to get the data back should they fail? I’ve developed my own solution to protect myself using some free and not-so-free tools from VMware, but I’m not sure it would scale well, or be easily manageable. Maybe a small challenge to the hardcore virtualizers out there may be in order. . . .

As soon as EMC’s announcement that it had added support for solid-state drives (SSDs) to Symmetrix crossed the wire, guess who called? If you’ve been watching the storage space, you know it had to be Hitachi Data Systems (HDS), whose high-end USP array has been do-si-doing around Symmetrix in the high-end disk array market for the last year and a half.

Turnabout’s fair play for HDS–as soon as it beat EMC to thin provisioning with the announcement of the USP-V last September, EMC went on the attack while both storage giants ignored the fact that they’d been soundly beaten to the feature by startups. I had a brief chat with HDS chief scientist Claus Mikkelsen yesterday, to see what HDS had to say about EMC’s blue-ribbon finish in the race to “tier zero.”

Generally when vendors gather to pooh-pooh one another’s products, they take one of two tacks: either poke holes in the soundness of the technology (EMC’s tactic in the earliest days of USP-V) or say there’s no market for it. In this case, HDS has taken the latter approach.

“Hitachi was in the solid-state disk business and the demand was very, very slight,” Mikkelsen began. Further questioning revealed that Hitachi’s disk division was offering standalone solid-state devices in the late 90’s…not quite the same business as flash drives embedded in an array, but I heard him out.

“Currently, flash has a limited number of writes before its memory layers wear out, and the use is limited to applications which are almost 100% very random reads,” he continued. “Even if EMC ships 10,000 solid state drives this year, it’s only .25 percent of their total shipments.”

Sour grapes? Maybe. “The drives they’re using have a SATA interface, you should be able to just pop them into any array,” Mikkelsen sniffed. “If they’ve created a market here, we’ll just jump right in.”

But out of curiosity, I also called a user for a major telecom which is a petabyte-plus EMC shop. This user and I have gotten into the nitty-gritty about performance-tuning storage before, and performance is king in his transaction-heavy environment. If this guy isn’t buying in, I thought, then who is?

Turns out he isn’t. “I think it’s great someone’s trying to make progress in this space–it’s been ignored,” he said. But even for his blue-chip company (he didn’t want it named in conjunction with his vendor), the whopping price tag for solid state drives is too much. “It has yet to get to the point where it’ll balance against savings on Tier 1 storage,” he said, though he admitted he has yet to do an in-depth analysis. “There might be certain cases…if we were less budget constricted or the timing was right, like we were going through a product refresh, we might look at it sooner, but for me this year, it’s not going to happen.”

EMC says it has the solid-state drives in beta tests in several of its “household name” customers’ shops. But did those shops pay for the drives? Did they pay full price? Will they put them into production? We don’t know for right now–EMC says they’re not available for interviews.

Even my friends who don’t normally follow the storage business are atwitter over an Engadget report that Buffalo has unleashed a 100 GB behemoth flash drive upon the world. Geeks everywhere are probably salivating to take the thing apart (yes, I’m looking at you, Tory) … unfortunately, they’ll have to wait. The catch is that Buffalo is only releasing the product for now in its home country of Japan.

According to company reps, the $1,000 asking price for the credit-card sized USB accessory makes it less than cost-effective to import right now. (If you just can’t get enough flash memory, there are 64 GB monsters roaming North America.)

The Engadget comments section also contains an interesting discussion of the merits of such a large flash drive. In the Engadget screenshot, the card looks like a  behemoth, but the post says it’s about the size of a credit card. Still, it launched a spirited discussion that I think asks some pertinent questions, namely, “would it not be more practical to just buy a $300 travel drive?”

At this juncture, and at this price point, certainly. But Moore’s law waits for no man, and the price of a 100 GB card will come down. Hence the other questions that this announcement begs: at what capacity and price point does a mechanical drive become more practical than a solid state drive? How will that equation change over time? It’s something we in the storage market are going to have to examine more closely in the coming year.

The glitz and glamour of new product releases tend to overshadow the rather mundane task of performing firmware upgrades on storage systems. However, administrators who take the time to keep their storage systems up-to-date with the latest and greatest patches for their storage system may find they can avoid some FC SAN “gotchas” as well as find some hidden gems that vendors are packaging in their latest firmware releases.

Prompting my thoughts on this topic was a recent conversation I had on this topic with a storage architect. He recently inherited a FC SAN where the firmware releases on the storage systems were two major releases back. The older code on these storage systems was becoming a problem since other devices on the SAN (switches, virtual tape libraries, and servers) had newer firmware with new features, but in order to take full advantage of these newer features, the storage systems also needed newer code.

I discussed this topic with EMC partly because the storage systems in question were EMC Clariion, but also because I know from personal experience that EMC releases firmware updates a fairly regular basis.

In the case of its Clariions, EMC comes out with a major release every 9 to 12 months that includes major new functions. For instance, its December 2006 code release for the Clariion included a new proactive hot spare feature for improved high availability and a Quality of Service feature as a licensable add-on. Its August 2007 Clariion major release added new security features as well as iSCSI enhancements like native replication.

Another interesting feature included in the update is the Software Assistant. This tool scans the Clariion prior to starting a firmware upgrade and provides recommendations as to which code an administrator should load on the system. The Software Assistant also does a high availability check prior to actually starting the upgrade to confirm that firmware upgrade can be completed without unexpectedly taking the system offline.

EMC recommends to customers that they install major firmware releases for its Clariions shortly after they are released (within 3 to 4 months).

However, there is a more pressing reason to ensure that firmware code is current. When doing firmware upgrades one must apply them sequentially.  If a Clariion system is two generations old, customers may need to upgrade to the intermediate release before upgrading to the newest release. Though this is generally not a big deal, it does add to the length of the time needed to perform the firmware upgrade and makes it more difficult to back out of an upgrade should something go awry.

In doing some research recently on the problems associated with recovering data from old tapes, I found out that a similar set of problems exist when trying to recover data stored on old disks. This problem becomes especially pronounced if a company unplugs an old disk drive and puts it on the shelf or keeps it in production too long.

The problem that companies are more likely to encounter when storing a disk drive on the shelf is not necessarily data degradation on the disk drive platter but mechanical failures of the parts within the disk drive itself. Greg Schulz, the lead analyst with Minneapolis-based StorageIO, finds that the lubricants of the mechanical parts within the disk drive can settle. This can cause the disk drive to malfunction when the company attempts power it up again for the first time in a long time.

Jim Reinert, VP of disaster recovery for Kroll Ontrack, a worldwide provider of data recovery services, says that the largest problem Kroll encounters with trying to recover data from old disk drives is repairing and replacing defective mechanical parts inside the disk drive. Motors failing and electronic circuit boards going bad are just some of the components Kroll has had to repair before it can recover the data from the drive. This situation requires Kroll to find an exact match for the defective part, usually on the used market.

Of course, mechanical problems can also occur while the computer system is still in use. Reinert finds that some of the toughest data to recover is found on older, proprietary computer systems that are in use but break. Typically found in manufacturing and production environments, these are older computer systems that control a piece of equipment that everyone uses but no one manages. As a result, the data is not backed up nor does anyone know who created the application or how it runs.

So, what’s the best way to protect data on old disk drives? The best and simplest way is to avoid keeping data on old disk drives and migrate data to newer disk drives. Kroll Ontrack classifies disk drives over five years in age as “old” since by this time disk drive warranties have usually expired and parts for the disk drive are out of production.

Schulz is a little less dogmatic about the five year cut-off. He finds that disk drives that are up to seven to eight years in age are probably OK depending on what condition in which they were stored or how they are used in production. He suggests spinning them up on a regular basis (once every 3 to 6 months), though he agrees that as disk drives age, administrators should migrate the data to newer drives.

If a disk drive has already failed or you come across one of indeterminate age or condition and you don’t know what data is on it or its value to your business, your best bet is probably to send it to a data recovery specialist and keep your fingers crossed.

Hitachi GST is back at it again this week with another update to its disk drives, this time with a redesign of its desktop SATA and PATA drives for power efficiency. Hitachi claims the updates to its silicon on the new Deskstar P7K500 drive can reduce the drive’s power consumption by up to 40 percent–or down to as low as 6 watts when active and 2 to 3 watts while idle.

The new specs were accomplished in a couple of different ways, one of which is the use of a new system on a chip model for power modules, and changing the power regulator on each drive from a linear architecture to a switched one. The moves were made with the new Energy Star 4.0 spec for PCs released in July, which allots a “budget” of 50 watts in idle mode for the whole system while idle, of which it’s estimated 8.3 go to the disk drive. With the new 250 GB version of the Deskstar, Hitachi is claiming a draw of 3.6 watts in idle mode, and 4.8 watts for the 320, 400 and 500 GB models.

This savings won’t necessarily make a dent in anyone’s home electric bill, according to Lee Johnson, 3.5-inch Product Marketing Manager for Hitachi. “But with the additional watts left over, PC makers can use that added wiggle room to design PCs with more RAM, more features on the motherboard, or a higher processor clock speed,” she said.

Hitachi plans to add similar power-savings technology to its enterprise-class drives, but IDC’s John Rydning says that may not necessarily be practical–nor lead to significant cost savings in enterprise disk systems.

“At the enterprise level there’s not a lot of impact on the overall system by reducing idle drive power draws,” he said, noting that turning drives completely off through MAID is the way the enterprise is headed. “But if you’re a large enterprise organization with hundreds or thousands of PC workstations, this might make a difference.”

Not sure if I mentioned this before, but I’m a geek. I like blinking lights and shiny things. I do math and physics for fun. I’d chose a good computer magazine over Maxim. . .well, maybe not THAT much of a geek, but you get the point.

So what’s provoked my geekitude this time? SAS benchmarks!

My friend Karl and I go back and forth about SAS disk benchmarks. I follow him in his quest to get past the 200MBps ceiling on his desktop. I poke fun at his pursuit while secretly hoping he’ll find that right combo to break the 200MBps mark so I can buy it.

Further fueling my mental yoga over disks is the fact that SAS has invaded our server room at work like a plague. A good plague, but a plague all the same. I went to work one day and realized we don’t use SCSI in anything but our older legacy machines. Honestly, I love it, the performance of SAS drives is great, they are small (we use 2.5-inch SAS on IBM blades) and they don’t make as much noise or heat, don’t use as much electricity and have a reasonable capacity.

So what’s the problem? The problem is that I go home (well, sometimes, anyway) and I don’t have SAS at home, I have SATA.

Mind you, my SATA array sits behind an Areca 8-port RAID controller with 128 MB of cache on a PCI-Express based card, so it’s no slouch. But it’s not SAS, not by a long shot.

I now.  . .must. . .have it. I neeeeeeeeeeed it. I don’t care what body part it’ll cost me! I want the speed and lightning response I get when I click the start menu or do some data migration chore on a SAS-based machine.

Vendors are now offering SAS cards with no RAID 5 or write cache available for about $150. The drives are about $250, which makes a small array at home not out of the question. (I just have to come up with a compelling argument to submit to the home finance committee. BTW, consider this an official cry for help to come up with an argument that will avoid the dreaded giant red “Denied–resubmit in 90 days” stamp the chair of said home finance committee has in her possession.)

But while trying to come up with this argument, it hit me. Traditional SCSI is dead as a doornail, and I missed the funeral.

In the meantime, if SATA ever slows down in its capacity growth, it had better look out too.

If I’m willing to sacrifice a bit of space for the speed, who else out there is willing to do the same? A decent capacity SATA disk will run you $200;  a 150 GB Western Digital Raptor (10k rpm SATA) will run you $220. So why bother? Why not spend the extra $30 and get SAS? The controllers are about the same cost now for quality brands, the cabling and power envelope are roughly the same, acoustics on the 2.5-inch drives are not bad and the thermal footprint is not outrageous.

And there’s a downside to size. How long would it take to rebuild a RAID 5 or 6 array made up of 4 TB drives ? How would I cope if I lost 4 TB of data?

My future holds a 32 GB to 64 GB RAID 1 solid-state disk for my OS, with capacity SAS for the 3 TB that Office 2010 is going to take up. IBM has already released a 16 GB SSD for their blades with the 32 GB models soon to be widely available. Not only that, but you can set them up in RAID 1. (Every time I say “RAID 1 SSD” I have to giggle.)

Can someone give me an irrefrangible (Thanks for the SAT submission! More more!!!) argument why SAS will not someday soon be the SATA of today?