Introduction and Internals
Could the Western Digital’s new Red be in your next NAS?
Introduction:
I'm going to let the cat out of the bag right here and now. Everyone's home RAID is likely an accident waiting to happen. If you're using regular consumer drives in a large array, there are some very simple (and likely) scenarios that can cause it to completely fail. I'm guilty of operating under this same false hope – I have an 8-drive array of 3TB WD Caviar Greens in a RAID-5. For those uninitiated, RAID-5 is where one drive worth of capacity is volunteered for use as parity data, which is distributed amongst all drives in the array. This trick allows for no data loss in the case where a single drive fails. The RAID controller can simply figure out the missing data by running the extra parity through the same formula that created it. This is called redundancy, but I propose that it's not.
Continue on for our full review of the solution to this not-yet-fully-described problem!
Since I'm also guilty here with my huge array of Caviar Greens, let me also say that every few weeks I have a batch job that reads *all* data from that array. Why on earth would I need to occasionally and repeatedly read 21TB of data from something that should already be super reliable? Here's the failure scenario for what might happen to me if I didn't:
- Array starts off operating as normal, but drive 3 has a bad sector that cropped up a few months back. This has gone unnoticed because the bad sector was part of a rarely accessed file.
- During operation, drive 1 encounters a new bad sector.
- Since drive 1 is a consumer drive it goes into a retry loop, repeatedly attempting to read and correct the bad sector.
- The RAID controller exceeds its timeout threshold waiting on drive 1 and marks it offline.
- Array is now in degraded status with drive 1 marked as failed.
- User replaces drive 1. RAID controller initiates rebuild using parity data from the other drives.
- During rebuild, RAID controller encounters the bad sector on drive 3.
- Since drive 3 is a consumer drive it goes into a retry loop, repeatedly attempting to read and correct the bad sector.
- The RAID controller exceeds its timeout threshold waiting on drive 3 and marks it offline.
- Rebuild fails.
At this point the way forward varies from controller to controller, but the long and short of it is that the data is at extreme risk of loss. There are ways to get it all back (most likely without that one bad sector on drive 3), but none of them are particularly easy. Now you may be asking yourself how enterprises run huge RAIDs and don't see this sort of problem? The answer is Time Limited Error Recovery – where the hard drive assumes it is part of an array, assumes there is redundancy, and is not afraid to quickly tell the host controller that it just can't complete the current I/O request. Here's how that scenario would have played out if the drives implemented some form of TLER:
- Array starts off operating as normal, but drive 3 has developed a bad sector several weeks ago. This went unnoticed because the bad sector was part of a rarely accessed file.
- During operation, drive 1 encounters a new bad sector.
- Drive 1 makes a few read attempts and then reports a CRC error to the RAID controller.
- The RAID controller maps out the bad sector, locating it elsewhere on the drive. The missing sector is rebuilt using parity data from the other drives in the array.
- Array continues normal operation, with the error added to its event log.
The above scenario is what would play out with an Areca RAID controller (I've verified this personally). Other controllers may behave differently. A controller unable to do a bad sector remap might have just marked drive 1 as bad, but the key is that the rebuild would be much less likely to fail as drive 3 would not drop completely offline once the controller ran into the additional bad sector. The moral of this story is that typical consumer grade drives have data error timeouts that are far longer than the drive offline timeout of typical RAID controllers, and without some form of TLER, two bad sectors (totaling 1024 bytes) is all that's required to put multiple terabytes of data in grave danger.
Update: These are now for sale on Newegg.com!
The Solution:
The solution should be simple – just get some drives with TLER. The problem is that until now those were prohibitively expensive. Enterprise drives have all sorts of added features like accelerometers and pressure sensors to compensate for sliding in and out of a server rack while operating, as well as dealing with rapid pressure changes that take place when the server room door opens and the forced air circulation takes a quick detour. Those features just aren't needed in that home NAS sitting on your bookshelf. What *is* needed is a Caviar Green with TLER, and Western Digital aims to deliver that, among other things:
For this review I have assembled all of the usual suspects that I've seen folks put into their home arrays (myself included). We have all three flavors of the Caviar Green (6Gb/sec, 3Gb/sec, and the AV-specific one meant for DVRs). I've also tossed in the Caviar Black and for the rich folks out there the enterprise grade RE4-GP and its faster spinning RE4 cousin. The 1TB VelociRaptor is included to give us some high-end perspective.
The Dirty Dozen (well, 2/3rd of it anyway).
Internals:
Sorry I won't be cracking this guy completely open, but here are some shots of what was accessible, including the back of the PCB:
A fast 6Gb/sec Marvell controller is assisted by 64MB of Samsung DDR2 cache. This should give us some good burst figures for sure.
Moving right along…>
put that in a Adrobo and i
put that in a Adrobo and i think you be good to go.
Actually, do we know whether
Actually, do we know whether Drobos are or are-not affected by a drive dropping out for “too long?”
I ask because the blurb on their website for their “BeyondRAID” technology say: “Built on an advanced virtualization platform…” [<-- this tells me nothing] and also "Since the technology works at the block level, it can write blocks of data that alternate between data protection approaches." My thinking here is that Drobos seem to be really smart about not exploding their own data, even on cheap and buggy drives. (Also, it would be nice if someone tells me I don't have to suddenly start worrying about this exact problem on the several Drobos I am acquainted with...) [edited for poor grammar]
You’re talking about the TLER
You’re talking about the TLER issue non-enterprise wd drives had while in raid setups…. I don’t believe it’ll be an issue here.
DROBO is NOT MAGIC. The
DROBO is NOT MAGIC. The quality of your drives CLEARLY counts.
I have used WD 1.5GB/2.0TB Greens in my 5 bay drobo and twice now in two years one of the drives has failed corrupting my data. Almost identical failures, the drive has bad sectors but its SMART reporting says its perfectly healthy. DROBO still thinks the drives are healthy, only thing that alerts you is on a reboot Win7 says your partition has a few unrecoverable files and it cannot fix them when it tries. Only way to find out ~which~ is the bad drive is to run WD’s diagnostic tools for many hours per drive and sure enough you’ll find the bad one in the stack. Replace it and then restore from backup – getting tired of this and its what I spent big bucks over an internal raid to try to avoid.
Backup, backup, backup.
Now that I know this stuff about TLER etc, I just ordered 5 WD Reds as a complete replacement slide em in and we’ll see how they do. Thanks for the great article Allyn.
Totally agree here. While the
Totally agree here. While the Drobo is very flexible with migrations, rebuilds, etc, the Drobo treats drive timeouts just like any other high end RAID system, and will benefit just as greatly from TLER-enabled drives.
It depends on how BeyondRAID
It depends on how BeyondRAID works, and that I am not sure of.
It is worth mentioning that software raid solutions (MDADM and ZFS primarily)aren’t subject to this issue. In the case a drive is taking to long to recover they will just cancel the operation and resubmit it, and should that fail again it will rebuild the bad stripe.
Also, TLER is on the firmware for every drive WD makes, but ever since they locked the firmware you cannot enable it on the green drives. Dirty money-grubbing bastards. With that being said, the raid drives do have other features (rotational vibration compensation, floating head, feed forward reading and all of that). Because of this though, I will probably never buy a WD drive again. Seagate wins the price vs performance debate hands down.
But ZFS takes integrity
But ZFS takes integrity further with background checking of blocks for corruption, and repairing / remapping PROACTIVELY.
I have owned various
I have owned various odd-sized drives (mostly WD consumer line) that eventually exceeded their SMART thresholds & would instantly draw a flurry of errors when directly attached in FreeBSD or even Windows 7–yet Drobo Dashboard would give them a clean bill upon initialization in my 2nd gen “appliance.” Since discovering this lovely quirk of the good old storage robot’s mysterious workings I’ve been wary of keeping anything on it too long without a fresh offline backup.
I wish you guys would have
I wish you guys would have put a comparison of the blue drive as well. I think Blue version is very popular giving great prices and decent performance. I really would have liked to see this Red drive compared to the Blue
Are these drives using the
Are these drives using the 1TB platters? (i.e. Do you think there would be any big performance drops between the 2TB and 3TB models?)
Yes, these are 1TB/platter
Yes, these are 1TB/platter drives afaik. I'll ask Allyn for that second question. I'm guessing the 2TB would be a bit slower than the 3TB but whether its enough to really be noticeable I dunno.
The gouge I got from WD
The gouge I got from WD suggests all three capacities should be nearly identical in performance. All three use 1TB per platter – lower capacities just have less platters, but the same density, so yeah, performance should not vary wildly among them.
TLER used to be something you
TLER used to be something you could enable on WD consumer drives. They removed this capability in newer drives. Do these dock their heads every few seconds like Green drives do/did back the last time I looked at platter drives?
I hate to be a cynic with nothing nice to say, but I’m not looking to do an enterprise datacenter RAID nor am I looking for a home server RAID. I’m looking for a workstation RAID-1 that is hardware failure resistant (and potentially with faster read speeds) and high performance (like my TLER enabled cav black 1TB drives). . .or just a single drive that’s backed with a long warranty and a data recovery guarantee.
I certainly don’t, I bought my last few drive purchases in pairs to RAID-1 against hardware failure with the understanding that if the volume failed to recover I’d be whipping out Recuva and other utilities to pick my files up.
How hard is it to physically take apart a drive and remove the platters? I’d like to see it done!
I believe the head dock time
I believe the head dock time has been extended to minimize repeated cycling.
I would recommend these over Greens for anything involving any sort of redundant RAID (your pairs included). I say this because the features are definitely worth it vs. the cost, and TLER is absolutely a good thing to have for those configurations. The flip side to this is if you also need superior random access performance for that same RAID, in which case I'd say go with VelociRaptors or even RE series drives (both also have TLER). Since TLER 'gives up' on bad sectors faster than a normal drive would, you should use them in redundant arrays. For single mass storage drives I'd say go Green, Blue, or Black – those will try harder to recover a bad sector, but that will only work properly outside of an array.
Finally – modern drives are *way* too dense and precise to open outside of a clean room. The last drive I did a head pack swap with was an 80GB unit. You can't do the steamy shower trick either :).
not only sleeps with them but
not only sleeps with them but also showers with them …nobody knows drives better than Allyn 🙂
> How hard is it to
> How hard is it to physically take apart a drive and remove the platters? I’d like to see it done!
If you have a set of TORX screwdrivers, it is very easy to take apart a drive and remove its platters.
Nevertheless, you must never do this, unless you intend to dump afterwards the HDD as garbage. A HDD cannot be put together again, after being taken apart, unless all it is done in a special clean room, with special tools.
Something I’ve yet to see in
Something I’ve yet to see in a review about these Red drives are comparisons to other manufacturers’ drives. Yes, WD only offers TLER on the RE and Red lines, but many manufacturers offer the same functionality on all their drives under the name CCTL. The warranty and MTBF is nice, but I’d rather have 7200rpm Barracudas with CCTL than these 5xxxrpm Reds. Especially in a RAID5 environment where random IO performance is important, spindle speed matters.
Could you post some details
Could you post some details of the script you run on your raid. I am sure there are plenty of us that already have a WD green drive raid and can’t afford to replace the whole raid in the short term.
Hi
From what I’ve heard and
Hi
From what I’ve heard and implemented myself, for large arrays, like the one mentioned, RAID-6 is recommended for increased security and indeed the increase likelihood of read-failure.
Regarding the read issues on specific drives, SMART should be used to continuously monitor the drive status: temperature, number of bad sectors, spin-up count, … Further SMART can and on large arrays should be configured for periodic tests:
* short self-tests
* long self-tests
* or partial long self-tests
This allowed me for example to detect recently a failing WD EARS 2g drive and replace in time.
Since TLER cause a timeout in
Since TLER cause a timeout in the recovery of a sector how does it affect the possible relocating of the sector in case it’s no good anymore? Won’t it prohibit the drive from finding out whether or not the sector is bad?
TLER just means the drive
TLER just means the drive considers the sector bad sooner than it would if it was a consumer drive. It won't be able to relocate the bad sector while it's bad, but it will know it's bad so hopefully the next time a write is attempted to the same spot it will then write to the spare area.
Note: for Advanced Format drives, logical sectors are part of larger physical sectors on the disk, so you're remapping 4K at a time, not 512B.
Allyn,
You mentioned that
Allyn,
You mentioned that you’ve been running a batch job to read the whole RAID array on a regular basis. That sounds like a good idea which I should do on my home NAS.
How are you accomplishing that feat? Do you get some type of a report that would tell you Disk 3 had a bad sector so you could deal with that before encountering a failure of Disk 1 (and subsequent RAID rebuild failure)? I assume that if you’re aware of this type of problem then simply trying to copy the file will trigger a problem for the RAID controller? And in the case of the Areca controller it should automatically recover the problem for you?
Do you know if the integrated RAID controllers on Intel and AMD motherboards also copy bad sectors the way you mentioned Areca handles this problem?
One thing that was unclear to me about RAID5 … does one bad sector on any disk guarantee a RAID rebuild failure? Shouldn’t it be possible to rebuild the rest of the RAID array and simply have one corrupted file instead of losing the whole RAID array?
For several years WD had been recommending not using their Green drives in RAID arrays. Recently I saw they okayed their use in 2 drive (RAID0 or RAID1) configurations. Given WD recommendation I have avoided using them in RAID5.
Too bad WD didn’t make these Red drives with an Unrecoverable Read Error rate of 10^15 versus the typical 10^14 for a consumer drive. That seems to be one advantage that Samsung drives have. Many are rated at 10^15, but now that Seagate has taken over Samsung HDD not sure how long that will remain true.
Great article. Appreciated all your hard work compiling and sharing these results.
Yeah.. the scenario described for the win. 🙂
Yeah.. the scenario described is precisely why I don’t run any kind of hardware RAID on a home server.
Right now the flavor of choice is ZFS and RAID-Z. zpool scrub
This may be a stupid question
This may be a stupid question but here goes. Could regular use of some kind of hdd utility such as “spinrite” go some way to preventing problems( i say stupid because i don’t even know if such utilities are able to be ued with a raid set up.)?
You could use a utility like
You could use a utility like SpinRite, but you would have to take the RAID offline, run SpinRite on each drive individually, and then bring the RAID back online. I’m not sure I’d recommend this option.
You could try to run SpinRite on a RAID while it is online at level 2, and it would do something similar to reading the entire RAID, but SpinRite does not have true block level access to the drive itself like it expects when drives are in RAID (other than RAID 0)
With the controller logic
With the controller logic being on the *drive* side of modern drives, there are severe limits to what Spinrite can accomplish. It's a great tool for exercising the entire drive surface by reading / writing / etc. Spinrite is coded to not timeout so easily on read errors, so it has a much more likely chance of making it across the whole disk span and prompting bad sectors to be mapped out, etc. Attempting the same feat under Windows would likely cause the drive to disappear after Windows gives up trying to access it after repeated errors.
Yes, I’ve been obsessing
Yes, I’ve been obsessing about this “bit error” issue on my Synology RAID-5 setup for some weeks now.
I thought my setup was safe as long as there was no fire or theft … but it seems that bit rot can creep into your array without noticing it until it’s too late.
Me too, I’m using Caviar Greens because those Enterprise drives are just WAY too pricy for a consumer like me.
Those RED drives really seem like the answer for me.
I’m just wondering whether there is a difference when using a true hardware RAID Areca/promise controller and a NAS like Synology/QNAP ?
As Synology is more a software raid solution, I think they compensate for TLER on consumer drives, so the drive won’t be kicked out of the array. Is this correct?
Would RAID-6 offer any benefits above RAID-5 concerning these ‘bit errors’ ?
Until I can get my hands on some RED’s, I’m using another NAS in JBOD to make a weekly backup.
But I’d prefer a true solid which provides a “disk failure and bit-error proof“ setup…
RAID-6 should theoretically
RAID-6 should theoretically be better able to cope with bit rot. If bit rot was detected during a regular read, or preferably during a scheduled scan, it will have double the parity available. Remember that bit rot could just as easily corrupt the parity information as the data stored. With a dual parity setup it’s possible to compare all three solutions, and with any luck two of the answers will come up equal meaning that the third, dissimilar answer is corrupt.
Now all of that is theoretical and only one possibility of how it can be handled by the controller or RAID software. AFAIK each sector on the HDD has a CRC checksum that should allow the drive to detect most cases of bit rot at the root. And with big sectors such as the WD Advanced Format they can even correct some cases of bit errors on the fly.
All of this is assuming that the bit rot is caused by instabilities in the media, however much more data is actually lost to software bugs and user errors, and unfortunately there is no RAID level that will help protect data from PEBKAC or programming errors.
Hi Allyn,
Would using the WD
Hi Allyn,
Would using the WD red make sense in a non-RAID environment?
If you just wish to store data which is not regularly read or written to; just for on-line storage would the red be a good option?
Kind Regards
Simon Zerafa
WD needs a refresh and update
WD needs a refresh and update to their RE4’s badly.
How are software RAID setups
How are software RAID setups (like mdadm) affected by these features (or lack thereof)?
They aren’t. See my above
They aren’t. See my above reply for more information.
Software RAID is quickly becoming the defacto standard among home users for this reason.
Nevermind, I did some
Nevermind, I did some RTFMing, the lack of TLER does not seem to be a problem for mdadm, which matches my anecdotal experience.
Hi guys, I have a 4 drive
Hi guys, I have a 4 drive NAS, that’s configured as solo drives (Non-raid configuration). Is this new Red drives recommended for such a configuration? Or would it be better to just use Green drives, as my Nas isn’t configured as a RAID. Hope someone can enlighten me here. Thanks
front load washing machines
front load washing machines have been using this method to counter-balance your dirty ginch for years and years 😉
Quick question!
I’m not a
Quick question!
I’m not a raid expert, but the example scenario you outline pops a few questions into my head.
Example 1:
Drive 1 goes offline, and during rebuild another hidden bad sector pops up in Drive 3 and kills the rebuild.
Example 2:
Drive 1 goes offline because the TLER reports the bad sector to the RAID card, and quickly address the issue using parity data from somewhere else.
There seems to be three factors, TLER vs non TLER, how a raid card address a bad sector, and the chance another bad sector would appear during a rebuild. The article makes it seem as if the Areca controller saved the day.
I draw this conclusion because Example 1 indicates the array fails because the raid controller failed to address the bad sector on Drive 1. Leaving the chance that a bad sector could appear during a rebuild, aka Drive 3. Whereas Example 2 has the array function because the drive reported the bad sector and the raid card address the issue correctly.
So the question I have is what is the timeout duration of a consumer drive to a TLER equipped drive? Is it a matter of hours? or days?
The question stems from the fact that I have several consumer drives in RAID-5 on a Areca 1231ML. I’d consider a over time upgrade, but if a TLER vs non TLER drive report a bad sector within days of each other, the card I have should address it correctly?
Appreciate the help.
Interesting. I just orderd 8
Interesting. I just orderd 8 WD RED and I am looking at RAID-cards now. Areca seems to be one of the best or am i wrong? I want a hardware-RAID-card that can recover from “small” errors. RE4 are too pricy. I already have a QNAP639pro (6 WD black 1TB). That’s worked perfect since the beginning. They (6x WD black 1TB) have run for well over 2 years 24/7 now and I espect a failur.. So I am building a new server after repleasing the drives in my QNAP-NAS with WD RED and 2 for and other RAID. Then I build a second with 2012 Essencial server when it appears and use the NAS as a backup for the new server. But I dont know if I am going to use WD RED or the pricy RE4/RE4-GP..
There was and is a sure fire
There was and is a sure fire way to get rid of a men.
However, there are guys out there who will balk at reading these words, wondering if they fall into can t get a
girlfriend the fat category. It’s Christmas time It makes us look good because it shows us how much they are hurting us and how it related to others, the time when they are in bars and discos.
my blog post :: Veronica
Interesting. I just orderd 8
Interesting. I just orderd 8 WD RED and I am looking at RAID-cards now. Areca seems to be one of the best or am i wrong? I want a hardware-RAID-card that can recover from “small” errors. RE4 are too pricy. I already have a QNAP639pro (6 WD black 1TB). That’s worked perfect since the beginning. They (6x WD black 1TB) have run for well over 2 years 24/7 now and I espect a failur.. So I am building a new server after repleasing the drives in my QNAP-NAS with WD RED and 2 for and other RAID. Then I build a second with 2012 Essencial server when it appears and use the NAS as a backup for the new server. But I dont know if I am going to use WD RED or the pricy RE4/RE4-GP..