Introduction and Internals

Could the Western Digital’s new Red be in your next NAS?

Introduction:

I'm going to let the cat out of the bag right here and now. Everyone's home RAID is likely an accident waiting to happen. If you're using regular consumer drives in a large array, there are some very simple (and likely) scenarios that can cause it to completely fail. I'm guilty of operating under this same false hope – I have an 8-drive array of 3TB WD Caviar Greens in a RAID-5. For those uninitiated, RAID-5 is where one drive worth of capacity is volunteered for use as parity data, which is distributed amongst all drives in the array. This trick allows for no data loss in the case where a single drive fails. The RAID controller can simply figure out the missing data by running the extra parity through the same formula that created it. This is called redundancy, but I propose that it's not.

Continue on for our full review of the solution to this not-yet-fully-described problem!

Since I'm also guilty here with my huge array of Caviar Greens, let me also say that every few weeks I have a batch job that reads *all* data from that array. Why on earth would I need to occasionally and repeatedly read 21TB of data from something that should already be super reliable? Here's the failure scenario for what might happen to me if I didn't:

  • Array starts off operating as normal, but drive 3 has a bad sector that cropped up a few months back. This has gone unnoticed because the bad sector was part of a rarely accessed file.
  • During operation, drive 1 encounters a new bad sector.
  • Since drive 1 is a consumer drive it goes into a retry loop, repeatedly attempting to read and correct the bad sector.
  • The RAID controller exceeds its timeout threshold waiting on drive 1 and marks it offline.
  • Array is now in degraded status with drive 1 marked as failed.
  • User replaces drive 1. RAID controller initiates rebuild using parity data from the other drives.
  • During rebuild, RAID controller encounters the bad sector on drive 3.
  • Since drive 3 is a consumer drive it goes into a retry loop, repeatedly attempting to read and correct the bad sector.
  • The RAID controller exceeds its timeout threshold waiting on drive 3 and marks it offline.
  • Rebuild fails.

At this point the way forward varies from controller to controller, but the long and short of it is that the data is at extreme risk of loss. There are ways to get it all back (most likely without that one bad sector on drive 3), but none of them are particularly easy. Now you may be asking yourself how enterprises run huge RAIDs and don't see this sort of problem? The answer is Time Limited Error Recovery – where the hard drive assumes it is part of an array, assumes there is redundancy, and is not afraid to quickly tell the host controller that it just can't complete the current I/O request. Here's how that scenario would have played out if the drives implemented some form of TLER:

  • Array starts off operating as normal, but drive 3 has developed a bad sector several weeks ago. This went unnoticed because the bad sector was part of a rarely accessed file.
  • During operation, drive 1 encounters a new bad sector.
  • Drive 1 makes a few read attempts and then reports a CRC error to the RAID controller.
  • The RAID controller maps out the bad sector, locating it elsewhere on the drive. The missing sector is rebuilt using parity data from the other drives in the array.
  • Array continues normal operation, with the error added to its event log.

The above scenario is what would play out with an Areca RAID controller (I've verified this personally). Other controllers may behave differently. A controller unable to do a bad sector remap might have just marked drive 1 as bad, but the key is that the rebuild would be much less likely to fail as drive 3 would not drop completely offline once the controller ran into the additional bad sector. The moral of this story is that typical consumer grade drives have data error timeouts that are far longer than the drive offline timeout of typical RAID controllers, and without some form of TLER, two bad sectors (totaling 1024 bytes) is all that's required to put multiple terabytes of data in grave danger.

 

Update: These are now for sale on Newegg.com!

The Solution:

The solution should be simple – just get some drives with TLER. The problem is that until now those were prohibitively expensive. Enterprise drives have all sorts of added features like accelerometers and pressure sensors to compensate for sliding in and out of a server rack while operating, as well as dealing with rapid pressure changes that take place when the server room door opens and the forced air circulation takes a quick detour. Those features just aren't needed in that home NAS sitting on your bookshelf. What *is* needed is a Caviar Green with TLER, and Western Digital aims to deliver that, among other things:

120710-065527-939.jpg

For this review I have assembled all of the usual suspects that I've seen folks put into their home arrays (myself included). We have all three flavors of the Caviar Green (6Gb/sec, 3Gb/sec, and the AV-specific one meant for DVRs). I've also tossed in the Caviar Black and for the rich folks out there the enterprise grade RE4-GP and its faster spinning RE4 cousin. The 1TB VelociRaptor is included to give us some high-end perspective.

120711-022810-859.jpg

The Dirty Dozen (well, 2/3rd of it anyway).

Internals:

Sorry I won't be cracking this guy completely open, but here are some shots of what was accessible, including the back of the PCB:

120711-031237-49.jpg

120711-030731-48.jpg

120711-030913-1075.jpg

A fast 6Gb/sec Marvell controller is assisted by 64MB of Samsung DDR2 cache. This should give us some good burst figures for sure.

Moving right along…>

 

« PreviousNext »