Latency Weighted Percentile – Intro and Individual Results
Our exclusive Latency Distribution / Latency Percentile testing was a long time in the making and was first introduced in my 950 Pro review (longer explanation at that link). To put it briefly, the thing that contributes greatest to the 'feel' of storage device speed is its latency. Simple average and maximum latencies don't paint nearly the full picture when it comes to the true performance of a given SSD. Stutters of only a few IO's out of the thousands delivered per second can simply be 'lost in the average'. This applies even if the average is plotted every second. The only true solution is to track the latency of each and every IO, no small feat when there are potentially hundreds of thousands (or millions) of IO's being delivered by the fastest SSDs.
Latency Distribution (V1)
Here the data has been converted into what is essentially a spectrum analyzer for IO latency. The more IO's taking place at lower latencies (towards the left of the 'spectrum') the better. While it is handy for seeing exactly where latencies fall for a given device, the results are generally hard to read and digest, so the data is further translated into a percentile:
Latency Percentile Plot (V1)
For those unfamiliar with this plot, the ideal result is a vertical line as far to the left as possible. Real world storage devices under load will tend to slant or slope, and some will 'turn' prior to hitting 100%, indicating that some of the IO's are taking longer (the point where the line curves back upwards indicates the latency of those remaining IO's).
This new testing has come a long way since it was first introduced. The most recent and significant change is to correct a glaring issue common to all IO percentile plots, caused by a bad assumption similar to that which comes with using averages. V1 Percentiles were calculated from the percentage of total IOs, which was in-line with what the rest of the industry has settled on. You might have seen enterprise SDS ratings claiming 99.99th (or some other variation e.g. 99.9% / 99.999%) percentile latency figures. As an example, a 99.99 percentile rating of 6ms would mean that 99.99% of all IOs were <= 6ms.
There is a flaw inherent in the above rating method. Using the 99.99% <= 6ms example above, imagine an SSD that completely stalled for one second in the middle of a 6-second run. For the other five seconds of the test, it performed at 200k IOPS. The resulting data would reflect one million total IO's and (assuming QD=1) a single IO taking a full second. The average IOPS would still be a decent 167k, but that nasty stutter was diluted – effectively 'lost in the average'. The same goes for 99.99% ("four nines") latency, which would miss that single IO. Despite hanging the entire system for 17% of the run, that single IO would not get caught unless you calculated out to 99.9999% ("six nines"), which nobody rates for.
The industry has settled on calculating this way mainly out of necessity and the limits of latency measurement. Most tools employ a coarse bucket scheme, meaning 99.99% values must be interpolated. Fortunately, our data gathering technique gives us far greater resolution into the data, meaning not only can we minimize interpolation, we can do something previously impossible. Get away from IO-based percentages means we must time-weigh our Latency Percentile results by summing not just the IO's, but the time those IOs took to complete. When calculated this way, our hypothetical example above would show low latency only up to the 83% mark, where its result would ride that 83% line all the way to the one-second mark on the plot. With these percentiles now based on total time and not the unweighted sum of the IO's, we can more easily identify those intermittent stalls.
Latency Percentile – Time Weighted (V2)
I've created the above based on the new weighted method but using the same source data as the earlier V1 plot. This data was based on reads, which don't suffer from the same inconsistent latencies seen in SSD writes. Even with more consistent results, we can see a difference in the plotted data. The RevoDrive 350 (red line) doesn't quite make it past 99% as quickly as it did in the V1 plot, and some of the faster SSDs taper off a bit earlier as well. The three HDDs also saw an impact, as longer seeks take up more of the total time of the run. I'm going to change gears and get into the results now, but I will revisit this at the end of the drive-to-drive comparisons on the next page and show just how much of a difference the weighting made on writes.
Latency Percentile – Individual Results
The workload chosen for these tests consists of completely filling an SSD with sequential data, then applying 4k random writes to an 8GB span. This is not the same as 'full span writes', which is more of an enterprise workload. Instead, we are emulating more of a consumer type of workload where only 8GB of the drive randomly written (typically by system log files, registry, MFT, directory structure, etc). The following is a random read Queue Depth sweep (1-32) of that same area, which tests how quickly the OS would retrieve those previously written directory structures and registry files.
I was going with zoomed scales for all of these write runs, but I could not do so with the HyperX Predator. It's QD32 run had an interesting oddity. 111 out of the 11.5 million total IO's in that run (0.001%) fell at the 1/2 second mark (500ms). That is a significant issue, but if we were only charting unweighted IO's here, that plot line would have climbed to 99.999% way back at the 0.4ms mark, camouflaging the issue. Our weighting shows a line riding at 98% out to 0.5s, which is accurate, as ~2% of the time spent on this test was servicing those IO's.
All SSDs prior to this point are NVMe-based. The following two are AHCI-based. Note the QD=1 latency shifts to the right by a bit due to the added latency inherent with the added overhead of that protocol. This will be more apparent in the direct comparisons done on the next page.
Now onto what this same 8GB span looked like during writes:
While reads were not much to write home about (pun intended), writes are a completely different story when it comes to SSDs. A keen eye will note that each controller / flash pairing appears to have its own unique write signature, and they are all significantly different from each other. This is the case even for SSDs that have similar IOPS.
The RD400 showed an odd 'plateau' at the ~63% mark, which extended out all the way into ~40ms territory. Here is what that looked like in Task Manager:
Despite that issue, it did 'ramp up' to that 63% mark much cleaner than the 950 Pro (below), enabling it to turn in higher average IOPS figures. Remember – roughly 1/3 of the time spent writing was waiting on IO's that were taking nearly 50ms to complete (HDD territory).
The 950 Pro is a solid performer in writes, but keep on reading – it is not the clear winner here.
The Kingston HyperX Predator performs fairly well here, but tail latencies do stretch out past 10ms.
The Intel SSD 750 is just a powerhouse when it comes to random write performance. This SSD is rooted in enterprise applications where Intel was *extremely* Gung-ho on keeping write latency as low and as consistent as possible, and they clearly succeeded as is demonstrated here. QD=1 latency is so low that I almost need to expand the low-end range of the chart!
There's no easy way to say this. The M6e is a hot mess when it comes to our latency testing. The IOPS figures are low as it is, and the latency profiles clearly demonstrate why.
Now look at that. This is *not* an NVMe SSD. It is a *SATA* SSD. Just look at how consistent that thing is! The throughput is almost immediately bottlenecked by its much slower interface, which explains the clean and even shifts to the right from QD=4 up. IOPS also remains identical starting at that same QD=4. The reason the line can shift to the right but result in the same IOPS is that each doubling of QD means there are double the IO's "in flight", and each IO must then wait twice as long to be serviced. The log scale is responsible for the shifts to appear even. As a descendent of a long line of SATA controllers, the 850 EVO is clearly the epitome of consistency optimization here, despite the slower bus.
Glad to see more competition
Glad to see more competition in this market, but gosh darn it those prices are still way too high.
You may be able to find the
You may be able to find the OEM version for less: the Toshiba XG3.
Am I wrong in thinking
Am I wrong in thinking this is the first consumer M.2 (without adapter), MLC, NVMe, drive to offer anything above 512GB? And, I thought I’d read somewhere that Samsung was to have released 1TB+ version of the 950 Pro Series by this Spring (which hasn’t happened).
Do you think Intel’s U.2/PCI 750 Series NVMe drives will start back down in price since they’d gone back up around $300.00 each for the 1.2TB version?
Good overall performance.
First consumer M.2 available in 1TB capacity.
Available with or without M.2 to PCIe adapter.”
Yeah OCZ got there first,
Yeah OCZ got there first, though Samsung should be ready to pull the trigger on launching their 48-layer VNAND 950 Pro. I suspect they were holding off due to lower demand at that capacity.
Actually Sandisk X400 drives
Actually Sandisk X400 drives launched couple of months back was the first consumer ssd with 1TB capacity in M.2 form factor.
I’m pretty sure the X400 is a
I’m pretty sure the X400 is a SATA III based AHCI drive and not NVMe.
I have a raid 0 of 840 evo
I have a raid 0 of 840 evo drives 2 500 GB where is the 2TB NVME drives at a reasonable cost. still see no point of upgrading.
Too small, but otherwise
Too small, but otherwise interesting.
The new weighted graphs are
The new weighted graphs are amazing, very educational.
Thanks! Lots of work has gone
Thanks! Lots of work has gone into their implementation and testing, but I'm still working out a better / cleaner way to present these massive amounts of data.
The graphs can get busy at
The graphs can get busy at times, and I personally think the only way to mitigate that might just be to make it interactive, (eg, the ability to show/hide data, they can overlap so much), something you might have been avoiding…
It shows how much effort you and the other editors put into these reviews. Whenever a new storage product launches I look for your review.
The new weighted graphs are
The new weighted graphs are amazing, very educational.
It is not so educational
It is not so educational without power consumption or temperature measures…
Remember that Flash drives are NVM so the heat is important for the memory retention, furthermore for high density chips!
It draws ~the same power as a
It draws ~the same power as a 950 Pro (~6 watts). With such a small draw with such a small surface area, temperatures vary *wildly* based on installation. If airflow is an issue, a single thermal pad to any adjacent component is more than enough to prevent possible thermal throttling.
Higher temps impact retention, but higher temps also make flash programming less damaging, meaning it is actually better to run your SSDs on the warmer side (to minimize wear effects) and store them at cool temperatures (to minimize leakage).
Higher temperature usually
Higher temperature usually makes material less resistive thus requiring less energy to write on Flash memory/PCM/etc but it also makes the storage less reliable.
You can’t just “store” an SSD at cool temperature when operating in read/write cycles…
Actually the Curie
Actually the Curie temperature of alloys used to make HDD platters is so high that it makes any HDD more reliable than the best SSD on the market.
I prefer a good old HDD
I prefer a good old HDD because its reliability cost less than the electronic Joule effect on SSD…
I’m still waiting WDC to fill its Velociraptor with Helium!
However I think WDC overpaid SNDK since the future of NVM would be the PCM technology and this would make obsolete the Flash memory.
So instead of using fast SSD
So instead of using fast SSD storage and having a backup plan you choose HDD because in your mind it’s more reliable (yet still fails)? Tell us how that order of magnitude higher access time is working out for you?
Fast failure isn’t worth the
Fast failure isn’t worth the extra pennies for SSD. 😉
If you consider to rebuild your system twice a month to be a feature then the SSD is for you…
From the maintenance POV I consider the MTTR and for my use the short access time and the built-in obsolesence of SSD (or Flash memory) aren’t profitable compared to the overpriced ticket.
Having to rebuild a desktop
Having to rebuild a desktop every 15 days on account of SSD failure? You’re doing it wrong.
I propose you resist, cause
I propose you resist, cause if you try an ssd once you’re addicted and can never go back. There is no ssd detox program.
Edit:reply was supposed to be one level up
That’s what I’ve told people
That's what I've told people since the X25-M – if you're not ready to buy one, don't sit at an SSD-equipped PC, because once you do, you'll want one *yesterday*.
Boy, that’s the truth!
Boy, that’s the truth!
That is absolutely correct.
That is absolutely correct. I got an SSD at my work computer several years ago…I couldnt stand to use my home PC’s anymore. I eventually had to go through and upgrade one by one my home PC’s to SSD. Then at work I got a 4K monitor….I couldnt stand to look at the “fuzzy” text at 1080p any longer, had to bite the bullet and upgrade one of my home PC’s to 4K (which required a new graphics card too).
That’s why I never stated to
That’s why I never stated to smoke sh!t… because it could damage my brain. :o)
I purchased my first SSD 4
I purchased my first SSD 4 years ago and have a total of 5 that I currently own. Have another 20-25 in builds for other people. Only failure was due to a lightning storm. Once you use ssd’s you cannot use a mechanical drive anymore except for mass storage. I am probably a year away from my NAS to be all ssd’s also.
A total of 5 SSD (for
A total of 5 SSD (for operating systems) in 4 years and you pretend this to be reliable…
I assume you own a home supercomputer. :o)
He probably has a RAID or is
He probably has a RAID or is using more of them in additional systems.
/Cant tell if troll or just
/Cant tell if troll or just terrified of SSD’s
I have three SSD’s in my desktop. Doesn’t mean two of them failed for me to get there.
So tell me why do you need so
So tell me why do you need so many unreliable data storages? :o)
One PCI-E Flash drive should be enough to win the top speed race!
Great, but where are the real
Great, but where are the real world benchmarks?
How much faster does it load battlefield?
Speed up my photo editing?
How long does it take to get into windows?
Its great I can copy a file to it at 1.5gps but where am I getting that file from?
Too bad, an SSD could write
Too bad, an SSD could write fast big data (movie capture for example) but it’s overpriced per Gigabyte and unreliable due to overheating chips…
I would probably still just
I would probably still just buy a Samsung 850 evo at the moment. I doubt most people would notice that much difference between an 850 evo and one of the pci-e, NVMe drives. Even if you do notice the difference, how much are you willing to pay for it? The difference in real world feel between a hard drive and a cheap SATA ssd is much larger than the jump from a SATA ssd to a pci-e, NVMe ssd.
There are a few applications where the pci-e, NVMe drive may be preferred though. Most consumer applications are not IO bound at all, but some professional applications may be. It would be nice to get opinions from people who use these SSDs in such a manner, but usually the best solution for such applications is to just use a lot of RAM and maybe a RAID array if you need the bandwidth. If you push a machine into using swap space for the working set, it will still slow to a crawl, even if you are swapping to an SSD and not a hard drive. For most consumers, these SSDs are in the high-end range where you pay a lot extra for a small performance gain. They do perform significantly better than SATA devices in benchmarks, but it just isn’t going to make that much of a difference for most consumer applications.
That’s the point of the
That's the point of the latency percentile testing. If the plots stretch across to the right and act like HDDs, then the system will feel more like a HDD for that remaining percent of the time. Boot times are impacted by too many other items unrelated to storage to matter – a clean windows install boots within a second delta between SATA SSD and NVMe, compared to the other 10-20 seconds it takes to get through UEFI/BIOS. The bigger thing that these NVMe drives can handle better is heavier multitasking workloads. Individual application tests don't really demonstrate this – it's when you've got multiple things going on where they really start to shine. If there was a consistent mixed simultaneous workload test then I would use it. PCMark doesn't cut it for the same reason we shifted to Latency Percentile testing – SSD stutters that would be painfully obvious to an actively engaged user are simply lost in the average of the trace test results.
Oh yeah that whole 'where are you gonna get that file from' is a mantra of mine, but the increased speed does translate to reduced latency (provided the controller is good, which is why we test that in such detail). Reduced latency translates to an overall snappier system, especially if the controller is mature and latencies are not only low but consistently low for more of the time and even under load.
Well Toms hardware did do
Well Toms hardware did do some realworld of the 850 vs 950 pro. I put their numbers in excel from some perspective.
World of Warcraft 0.88% Faster
BF3 1.38% Faster
Photoshop Light 3.48% Faster
Photoshop Heavy 3.38% Faster
Aftereffects 3.09% Faster
Adobe Illustrator 1.42% Faster
Word 1.44% Faster
Excel 1.11% Faster
Power Point 2.25% Faster
Average 2% faster……
2% for something that cost 50% to 300% more than a normal SSD. Better to put the money in CPU, GPU, or just plain more SSD space.
I get that under super heavy workload this will start to shine (maybe) but for 90% of enthusiasts let alone the average consumer money is better spent else where or not at all.
I know you often bring up that mantra on the podcast and I was quoting you a little when I brought it up. I would like to have you guys bring up these points I made in the podcast and see what everyone thinks.
Methinks PC Per File Copy
Methinks PC Per File Copy Test needs a few more files in it. Six of the contenders within 1.5 seconds of each other, maybe a longer test would give a better observable spread and minimise systematic errors.
The test is already pretty
The test is already pretty long and repeats 3x / averages the results, but I do see your point. I'm working on morphing this test into something that better shows the actual throughput seen in file creations and copies (with better tools).
Give Allyn a raise! Since
Give Allyn a raise! Since introducing the Latency percentile testing, pcper is the 1# place for SSD reviews and offers more insight about SSD-s than all other review sites combined.
Agreed! Always has been…
Agreed! Always has been…
Looks like really crappy
Looks like really crappy choice compared to 950pro – you are trading 40 (or was it 50) nm V-NAND MLC for 15nm planar NAND for few bucks savings.
Also while everyone is writing about 1TB SKU being advantage I’d say that 128GB is what makes this drive interesting since it makes very nice OS drive for few bucks
True, but remember that 128GB
True, but remember that 128GB will also have a fraction of the dies and should see decreased performance (especially on writes) due to the reduced parallelism.
Can these be used as boot
Can these be used as boot drives with older motherboards? What do they appear as to the bios?
You have to have a system /
You have to have a system / BIOS that supports NVMe boot. X99 should all have support, Z97 *might* support it, and Z170 should all have support.
The reason this is different than in the past is that NVMe devices carry their own driver which the UEFI/BIOS must be aware of and read prior to being able to initialize the NVMe SSD for boot.
Excellent work Allyn this new
Excellent work Allyn this new testing methodology keeps getting better.