Introduction
IOSP and GB/s does *not* tell the whole story!
NVMe was a great thing to happen to SSDs. The per-IO reduction in latency and CPU overhead was more than welcome, as PCIe SSDs were previously using the antiquated AHCI protocol, which was a carryover from the SATA HDD days. With NVMe came additional required support in Operating Systems and UEFI BIOS implementations. We did some crazy experiments with arrays of these new devices, but we were initially limited by the lack of native hardware-level RAID support to tie multiple PCIe devices together. The launch of the Z170 chipset saw a remedy to this, by including the ability to tie as many as three PCIe SSDs behind a chipset-configured array. The recent C600 server chipset also saw the addition of RSTe capability, expanding this functionality to enterprise devices like the Intel SSD P3608, which was actually a pair of SSDs on a single PCB.
Most Z170 motherboards have come with one or two M.2 slots, meaning that enthusiasts wanting to employ the 3x PCIe RAID made possible by this new chipset would have to get creative with the use of interposer / adapter boards (or use a combination of PCI and U.2 connected Intel SSD 750s). With the Samsung 950 Pro available, as well as the slew of other M.2 SSDs we saw at CES 2016, it’s safe to say that U.2 is going to push back into the enterprise sector, leaving M.2 as the choice for consumer motherboards moving forward. It was therefore only a matter of time before a triple-M.2 motherboard was launched, and that just recently happened – Behold the Gigabyte Z170X-SOC Force!
This new motherboard sits at the high end of Gigabyte’s lineup, with a water-capable VRM cooler and other premium features. We will be passing this board onto Morry for a full review, but this piece will be focusing on one section in particular:
I have to hand it to Gigabyte for this functional and elegant design choice. The space between the required four full length PCIe slots makes it look like it was chosen to fit M.2 SSDs in-between them. I should also note that it would be possible to use three U.2 adapters linked to three U.2 Intel SSD 750s, but native M.2 devices makes for a significantly more compact and consumer friendly package.
With the test system set up, let’s get right into it, shall we?
While interesting as an
While interesting as an exercise (I know only 3 ports) I find nothing short of sacrilege to suggest RAID5 setup on Flash based drives. If it was drivepool kind of setup then fine, but it’s not. NVMe doing great in R1 or 10. There is no point wearing NAND with unnecessary parity writes (just like classic SSDs). Basically if you value your SSD all parity based Raid levels are out of the window. Even in enterprise environment SSD parity arrays are rarely encountered. And all of that with SSDs which cost 10-30x more than consumer grade drives. Simply 1 or 10 is much more convenient and easier&faster to recover. Time is money.
Wearing out ssd’s is not that
Wearing out ssd’s is not that big an issue anymore, these vnand chips have excellent durability.
Modern SSD drives will run
Modern SSD drives will run for years just fine under heavy RAID5. Plus with RAID, the real issue with failure is effect on cost over time (because there is no data loss with a single drive failure). And RAID5 is extremely cost effective, because you can get to, say, 1TB of data with only 3 500GB drives and still have parity. That compares favorably for cost with using 2 1TB drives in RAID1. The slight reduction in life from the parity writing is nowhere close to enough to offset that cost savings. Therefore, RAID5 should be a recommended configuration where expensive SSD cost is a chief concern.
The biggest downside is that this board only supports 3 M.2 drives, the minimum (and least cost-effective per Byte) number of drives needed for RAID5. This also means no expansion capability, which is possible with SATA SSDs in RAID5.
nice review,but i want to
nice review,but i want to more real life test than grafs ans charts
I too would be curious about
I too would be curious about some real life tests. (windows boot and shutdown, app startup, etc)
Boot of a ‘clean’ fresh
Boot of a 'clean' fresh install is essentially the same (or in some cases it takes a second or two longer due to different initialization of some BIOS when initializing NVMe devices during boot). Where the speed difference would be more seen is a 'well used' OS that has had a lot of other apps / startup processes / cruft generated over time. The additional SSDs would keep latency lower during the increased load seen during that boot. Still, we are talking a few seconds time, and that only happens while booting, which is a rare event (and why we don't focus on that aspect).
Why cant M2 slots be at right
Why cant M2 slots be at right angles to the motherboard ?This would save space and allow better airflow and use less space. I could possibly see myself getting two m2 in raid0 at 120gb rather than a single 240gb.It would be interesting to see results of windows raid also, which has been flawless in my system (win7 ).
I think there was a few
I think there was a few boards that did that; maybe some ASUS board if I remember right. It is a little more expensive since you need a metal bracket to support it.
For RAID 5, can you test with
For RAID 5, can you test with a LSI controller? I’m sure that the onboard RAID controller will be the bottleneck calculating the parity.
LSI RAID controllers handle
LSI RAID controllers handle SAS / SATA. These are PCIe NVMe.
The ASRock Z170 OC Formula
The ASRock Z170 OC Formula and Extreme 7 both also have triple M.2 and are around half the price. They’ve been out for some time now. You should check them out to see if they have a same or worse RAID implementation.
The RAID implementation is in
The RAID implementation is in the Z170 chipset, so it should be exactly the same. It does have some hardware acceleration, but it doesn’t seem to have hardware parity calculations. It would be cool it they could make a PCIe x16 RAID card that can handle 4 of these SSDs. No home user needs such a thing though.
This specific board is probably really expensive sinxe it has a PLX chip to convert the x16 PCIe connections from the CPU out to x32. This allows for 4-way SLI with x8 PCIe to all 4 slots.
The only “wise” usage to
The only “wise” usage to explain the price is in a rack used for real time financial transaction (read write databases)
Not really, as that type of
Not really, as that type of use would be full span random writes, which would slow these consumer SSDs down considerably. You need enterprise optimized firmware for that.
In the video, you mention
In the video, you mention that “advanced” users could see up to QD=8 . But, everything I’ve read says most users don’t go past QD=2.
What exactly makes users go to higher queue depths? Like, is downloading a file + gaming going to increase your QD?
Each individual thread
Each individual thread (program) that hits the storage will add *at least* one to the QD figure. Apps can individually ask for multiple sectors at the same time, or can 'ask ahead', which builds the queue. A simple windows file copy can run at QD=4 with nothing else going on. QD can spike past 64 on a powerful, multi-core system during boot where dozens of other apps and services are simultaneously launching. Note: SATA devices can't exceed QD=32, so if the OS climbs higher, the queue backs up into the OS itself, no additional benefit will come from a SATA SSD (since it can't see further ahead than the next 32 requests).
Great job as usual Allyn,
Great job as usual Allyn, excellent info and methodology. I have one of the 950 pros and had to use an Asus Hyper X4 riser card on my X99 because the native M.2 are worthless 10G slots and that’s when it dawned on me that I wouldn’t be able to do RAID with M.2 because the rest of my PCIe slots are occupied.
What we REALLY need are PCIe x16 riser cards that can support up to FOUR M.2 2280 cards for either RAID or JBOD, but the most important thing is it consolidates space and slots which is a problem now due to the way M.2 are routed with HSIO since they need to work individually or in tandem.
Do you have any info on this from industry OEMs? An x16 riser card that could take 4x M.2 would be awesome!
> An x16 riser card that
> An x16 riser card that could take 4x M.2 would be awesome!
There certainly is a lot of engineering elegance to be had
with four M.2 @ x4 PCIe 3.0 lanes = x16 PCIe 3.0 lanes.
However, a PLX-type chip is required because
PCI-Express does not generally allow multiple
discrete devices in a single PCIe expansion slot.
HP and Dell have already developed same,
but the HP version requires an HP workstation.
For photos and discussion:
Google “Cheap NVMe performance from HP”
We published a WANT AD for same several months ago,
and one storage expert confirmed that h/w RAID
controllers are “works in progress” but
he was limited by an NDA and couldn’t say
much more. To locate our WANT AD:
Google “NVMe RAID controller”
How about motherboards that replace SATA-Express ports
with 4 x U.2 ports? There’s certainly enough room.
A factor to consider is the upstream bandwidth
of the DMI 3.0 link = 4.0 Gb/s (basically
x4 PCIe 3.0 lanes @ 8 GHz / 8.125 bits per byte).
As such, the upstream bandwidth of a single
NVMe M.2 connector is exactly the same
as the upstream bandwidth of the DMI 3.0 link.
It should be very interesting when Optane
(Intel 3D XPoint) non-volatile memory
becomes available in the M.2 form factor:
that development should create lots of pressure
to increase the upstream bandwidth to satisfy
that extra demand.
At the moment, barring any major changes in
Intel’s latest chipsets, RST and RSTe
will only work DOWNSTREAM of the DMI 3.0 link:
RST does NOT work with the x16 lanes controlled
directly by any Intel CPUs, as far as I know.
Allyn, if you’re reading this, could you
possibly confirm or update any of the above, please?
I would like to refine my understanding of these
issues, so as not to mislead anyone else.
GREAT REVIEW, once again!
Do keep up the brilliant work, Allyn.
You be the best, man 🙂
MRFS
To RAID NVMe devices the only
To RAID NVMe devices the only current games in town are:
There is an HP Z Turbo Drive that supports up to 4x M.2, but it is current not clear (and unlikely) if it is performing hardware RAID of NVMe devices.
FYI: here are the 3 add-in
FYI: here are the 3 add-in cards that
reportedly support 4 x M.2 SSDs
and an x16 edge connector:
HP Reveals New Z Turbo Drive Quad Pro
http://www.storagereview.com/hp_reveals_new_z_turbo_drive_quad_pro
The Dell 4x m.2 PCIe x16 version of the HP Z Turbo Quad Pro
http://www.servethehome.com/the-dell-4x-m-2-pcie-x16-version-of-the-hp-z-turbo-quad-pro/
Kingston Unveils E1000 NVMe Enterprise SSD At CES 2016
http://www.tomsitpro.com/articles/kingston-e1000-ssd-nvme-liqid,1-3098.html
Thanks again, Allyn.
MRFS
correction:
DMI 3.0 link =
correction:
DMI 3.0 link = 4.0 Gb/s
should be
DMI 3.0 link = ~4.0 GB/s (32 Gb/s / 8.125 bits per byte)
sorry for the typo
MRFS
Kingston has one
Kingston has one also:
http://www.tomsitpro.com/articles/kingston-e1000-ssd-nvme-liqid,1-3098.html
Allyn, One more thing:
Is
Allyn, One more thing:
Is it possible to install a high-performance SAS RAID
controller in the primary PCIe 3.0 slot, and
have it communicate directly with the CPU
rather than via the DMI 3.0 link?
I for one would be very interested in seeing
if such a configuration can circumvent
the 4.0 GB/s bandwidth ceiling of the DMI 3.0 link.
In theory, a PCIe 3.0 x8 edge connector
should have twice the bandwidth ceiling of DMI 3.0
e.g. x8 PCIe 3.0 lanes @ 8G / 8.125 = ~8 GB/s.
If Intel’s RSTe only works downstream of the DMI 3.0 link,
it seems that a 12 Gb/s SAS controller with 8 x 12G ports
should exceed the DMI ceiling e.g. by configuring a
RAID-0 with 12G SAS SSDs e.g. Toshiba PX04SL SSDs:
http://www.storagereview.com/toshiba_releases_new_readintensive_sas_ssd
Maybe Toshiba will lend PCPER some samples
so you can do a scaling experiment.
Now, what SAS RAID controller would be best,
Areca? LSI? ATTO? And, will such a controller
work in the primary PCIe 3.0 expansion slot?
Here’s an Avago model at Newegg:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118217&Tpk=N82E16816118217
MRFS
Yes, a PCIe x8 RAID card can
Yes, a PCIe x8 RAID card can exceed DMI 3.0 (4 lane) bandwidth, but you are adding a bunch of latency and a lower maximum cap on ultimate IOPS that the RAID controller can handle. Intel RST (SATA) actually beats most add-in RAID cards as far as IOPS scalability goes. It would also not be able to communicate to the host via NVMe, so there would be the same sort of IO overhead seen with SATA. It's basically the long / expensive way go reach those high figures.
Many thanks for your very
Many thanks for your very prompt replies above.
Keep up the great work, Allyn.
MRFS
Re: HP Z Turbo Quad Pro
Re: HP Z Turbo Quad Pro reportedly has a “BIOS lock”:
http://www.tomsitpro.com/articles/hp-reveals-turboz-quad-pro,1-3022.html
“Thanks to a BIOS lock, the device is supported only on the HP Z440, Z640, and Z840 Workstations, and cannot be used in any other OEM workstation solution.”
Would the Windows 10 Pro USB
Would the Windows 10 Pro USB flash drive work for OS installation with this configuration (is it GPT formatted)?
FOUND! PLX
FOUND! PLX Heaven:
http://www.servethehome.com/wp-content/uploads/2015/08/One-Stop-Systems-Avago-PCIe-switch-board-for-m2-SSDs.jpg
How about a workstation motherboard with
multiple U.2 ports, like this:
http://www.servethehome.com/wp-content/uploads/2015/08/A-Serial-Cables-Avago-PCIe-switch-board-for-NVMe-SSDs.jpg
Here’s the full article:
http://www.onestopsystems.com/blog-post/avago-and-plx-%E2%80%93-future-pcie
Avago + Malventano = exabytes per nanosecond! 🙂
MRFS
Here’s an add-on card with an
Here’s an add-on card with an x16 edge connector
and four U.2 ports:
http://www.serialcables.com/downloads/PCI-HBx16-I.pdf
I doubt that it supports hardware RAID, however:
note where it says “requires no additional software”.
The company is called Serial Cables:
http://www.serialcables.com/
Here’s the spec page for the
PCIe Gen3 Switch Board mentioned above:
http://www.serialcables.com/products.asp?cat=351&tier=264
Allyn, if you’re still reading this, does that
Switch-based Host Adapter appear very similar to the two
made by Supermicro?
It sure would be nice to have a workstation motherboard
with four U.2 ports, just like that Host Adapter.
MRFS
… replacing 2 x
… replacing 2 x SATA-Express ports,
like this:
http://supremelaw.org/systems/nvme/4xU.2.and.SATA-E.jpg
I tried to maintain the same scale:
if so, there’s plenty of room for 3 more U.2 ports
if we remove the SATA-Express ports.
MRFS
So to answer a quick
So to answer a quick question,
with three M.2 SSDs installed, will my graphics card run at x8?
Also to clarify this, I will have two SATA ports available for any optical or HDD or SSD to add further on?
Thanks a bunch.
Got my answer from the
Got my answer from the comments of the Video.
FYI: Icy Dock sent me this
FYI: Icy Dock sent me this email ad yesterday;
because the source is obvious, I’m sharing this
email message under the “fair use” doctrine:
http://supremelaw.org/systems/icydock/ICY.DOCK.MB998SP-B.in.HP.Z840.htm
Those Areca controllers are pricey:
http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Order=BESTMATCH&Description=Areca+ARC-1883ix&N=-1&isNodeId=1
I couldn’t find motherboard details for that
HP workstation: the Areca they used could not
have been downstream of the DMI link — NOT
with the performance numbers they reported.
-OR-
Are those numbers merely an effect of the
large on-board cache?
MRFS
I plan to build a high end
I plan to build a high end gaming computer for iRacing.com use. It seems to me that 3 m.2’s in Raid 0 are not going to increase my FPS, though it may help with load times from the sad’s to memory for tracks and cars.
Also, isn’t Raid 0 a bit unstable? If there is any sort of memory error on either SSD won’t that lock up the OS?
Even 2 SSD’s in Raid 0 doesn’t appear to add a significant advantage for my system.
Great discussion, article and video. I learned a great deal.
Thanks!