PC Perspective Custom SSD Test Suite Introduction
Back in late 2016, we implemented a radically new test methodology. I'd grown tired of making excuses for benchmarks not meshing well with some SSD controllers, and that matter was amplified significantly by recent SLC+TLC hybrid SSDs that can be very picky about their workloads and how they are applied. The complexity of these caching methods has effectively flipped the SSD testing ecosystem on its head. The vast majority of benchmarking software and test methodologies out there were developed based on non-hybrid SLC, MLC, or TLC SSDs. All of those types were very consistent once a given workload was applied to them for long enough to reach a steady state condition. Once an SSD was properly prepared for testing, it would give you the same results all day long. No so for these new hybrids. The dynamic nature of the various caching mechanisms at play wreaks havoc on modern tests. Even trace playback testing such as PCMark falter, as the playback of traces is typically done with idle gaps truncated to a smaller figure in the interest of accelerating the test. Caching SSDs rely on those same idle time gaps to flush their cache to higher capacity areas of their NAND. This mix up has resulted in products like the Intel SSD 600p, which bombed nearly all of the ‘legacy’ benchmarks yet did just fine once tested with a more realistic, spaced out workload.
To solve this, I needed a way to issue IO's to the SSD the same way that real-world scenarios do, and it needed to be in such a way that did not saturate the cache of hybrid SSDs. The answer, as it turned out, was staring me in the face.
Latency Percentile made its debut in October of 2015 (ironically, with the 950 PRO review), and those results have proven to be a gold mine that continues to yield nuggets as we mine the data even further. Weighing the results allowed us to better visualize and demonstrate stutter performance even when those stutters were small enough to be lost in more common tests that employ 1-second averages. Merged with a steady pacing of the IO stream, it can provide true Quality of Service comparisons between competing enterprise SSDs, as well as high-resolution industry-standard QoS of saturated workloads. Sub-second IO burst throughput rates of simultaneous mixed workloads can be determined by additional number crunching. It is this last part that is the key to the new test methodology.
The primary goal of this new test suite is to get the most accurate sampling of real-world SSD performance possible. This meant evaluating across more dimensions than any modern benchmark is capable of. Several thousand sample points are obtained, spanning various read/write mixes, queue depths, and even varying amounts of additional data stored on the SSD. To better quantify real-world performance of SSDs employing an SLC cache, many of the samples are obtained with a new method of intermittently bursting IO requests. Each of those thousands of samples is accompanied by per-IO latency distribution data, and a Latency Percentile is calculated (for those counting, we’re up to millions of data points now). The Latency Percentiles are in turn used to derive the true instantaneous throughput and/or IOPS for each respective data point. The bursts are repeated multiple times per sample, but each completes in less than a second, so even the per-second logging employed by some of the finer review sites out there just won’t cut it.
Would you like some data with your data? Believe it or not, this is a portion of an intermittent calculation step – the Latency Percentile data has already been significantly reduced by this stage.
Each of the many additional dimensions of data obtained by the suite is tempered by a weighting system. Analyzing trace captures of live systems revealed *very* low Queue Depth (QD) under even the most demanding power-user scenarios, which means some of these more realistic values are not going to turn in the same high queue depth ‘max’ figures seen in saturation testing. I’ve looked all over, and nothing outside of benchmarks maxes out the queue. Ever. The vast majority of applications never exceed QD=1, and most are not even capable of multi-threaded disk IO. Games typically allocate a single thread for background level loads. For the vast majority of scenarios, the only way to exceed QD=1 is to have multiple applications hitting the disk at the same time, but even then it is less likely that those multiple processes will be completely saturating a read or write thread simultaneously, meaning the SSD is *still* not exceeding QD=1 most of the time. I pushed a slower SATA SSD relatively hard, launching multiple apps simultaneously, trying downloads while launching large games, etc. IO trace captures performed during these operations revealed >98% of all disk IO falling within QD=4, with the majority at QD=1. Results from the new suite will contain a section showing a simple set of results that should very closely match the true real-world performance of the tested devices.
While the above pertains to random accesses, bulk file copies are a different story. To increase throughput, file copy routines typically employ some form of threaded buffering, but it’s not the type of buffering that you might think. I’ve observed copy operations running at QD=8 or in some cases QD=16 to a slower destination drive. The catch is that instead of running at a constant 8 or 16 simultaneous IO’s as you would see with a saturation benchmark, the operations repeatedly fill and empty the queue, meaning the queue is filled, allowed to empty, and only then filled again. This is not the same as a saturation benchmark, which would constantly add requests to meet the maximum specified depth. The resulting speeds are therefore not what you would see at QD=8, but actually, a mixture of all of the queue steps from one to eight.
Conditioning
Some manufacturers achieve unrealistic ‘max IOPS’ figures by running tests that place a small file on an otherwise empty drive, essentially testing in what is referred to fresh out of box (FOB) condition. This is entirely unrealistic, as even the relatively small number of files placed during an OS install is enough to drop performance considerably from the high figures seen with a FOB test.
On the flip side, when it comes to 4KB random tests, I disagree with tests that apply a random workload across the full span of the SSD. This is an enterprise-only workload that will never be seen in any sort of realistic client scenario. Even the heaviest power users are not going to hit every square inch of an SSD with random writes, and if they are, they should be investing in a datacenter SSD that is purpose-built for such a workload.
Calculation step showing full sweep of data taken at multiple amounts of fill.
So what’s the fairest preconditioning and testing scenario? I’ve spent the past several months working on that, and the conclusion I came to ended up matching Intel’s recommended client SSD conditioning pass, which is to completely fill the SSD sequentially, with the exception of an 8GB portion of the SSD meant solely for random access conditioning and tests. I add a bit of realism here by leaving ~16GB of space unallocated (even those with a full SSD will have *some* free space, after all). The randomly conditioned section only ever sees random, and the sequential section only ever sees sequential. This parallels the majority of real-world access. Registry hives, file tables, and other such areas typically see small random writes and small random reads. It’s fair to say that a given OS install ends up with ~8GB of such data. There are corner cases where files were randomly written and later sequentially read. Bittorrent is one example, but since those files are only laid down randomly on their first pass, background garbage collection should clean those up so that read performance will gradually shift towards sequential over time. Further, those writes are not as random as the more difficult workloads selected for our testing. I don't just fill the whole thing up right away though – I pause a few times along the way and resample *everything*, as you can see above.
Comparison of Saturated vs. Burst workloads applied to the Intel 600p. Note the write speeds match the rated speed of 560 MB/s when employing the Burst workload.
SSDs employing relatively slower TLC flash coupled with a faster SLC cache present problems for testing. Prolonged saturation tests that attempt to push the drive at full speeds for more than a few seconds will quickly fill the cache and result in some odd behavior depending on the cache implementation. Some SSDs pass all writes directly to the SLC even if that cache is full, resulting in a stuttery game of musical chairs as the controller scrambles, flushing SLC to TLC while still trying to accept additional writes from the host system. More refined implementations can put the cache on hold once full and simply shift incoming writes directly to the TLC. Some more complicated methods throw all of that away and dynamically change the modes of empty flash blocks or pages to whichever mode they deem appropriate. This method looks good on paper, but we’ve frequently seen it falter under heavier writes, where SLC areas must be cleared so those blocks can be flipped over to the higher capacity (yet slower) TLC mode. The new suite and Burst workloads give these SSDs adequate idle time to empty their cache, just as they would have in a typical system.
Apologies for the wall of text. Now onto the show!
OK so with all that data…is
OK so with all that data…is it faster than a 960 evo for everyday windows use? and does it use more or less power for a laptop?
Likely less power in laptop
Likely less power in laptop usage as it spends less time servicing IOs on average, but the 'feel' will likely be similar unless you are doing some heavy mixed/random read workloads.
Any idea why Intel chose the
Any idea why Intel chose the PCI-E 3.0 x2 interface? All M.2 slot that support this drive are x4 capable. Seems like they are leaving performance on the table….
On a related note, correct me if I’m wrong, but most Z370 motherboards have 20 PCI-E lanes available without going through the chipset’s 4 additional lanes. Usually they are assigning 16 lanes to the graphics card and 4 lanes to an M.2 slot. Additional M.2 slots are running through the chipset’s lanes. Are existing motherboards capable of assigning 2 CPU lanes to 2 different M.2 slots?
They likely went with x2
They likely went with x2 since they already had hardware/controller close to that ready to go (via Optane Memory parts).
Current X299 boards / VMDs can only bifurcate in x4 chunks, but someone can probably make a PCIe switch that can split further.
Re: ASUS Sabertooth
Re: ASUS Sabertooth X99
Allyn,
Would it make any sense to evaluate these Optane parts on an AMD Threadripper system?
My best guess is that your X99 test system was capable of driving both the M.2 and x16 slots at max speed, so an AMD motherboard should not make much of a difference.
Again, thanks for your consistently excellent reviews!
p.s. Did you ever solve the problems you encountered with your Threadripper system?
In short, it wouldn’t, not if
In short, it wouldn't, not if I wanted the lowest and most consistent latencies.
I've been sticking with the X99 platform because 1. If it ain't broke… and 2. I'd rather not re-test 100+ SSDs, not until I've added some workloads to the suite at least.
Re: problems on TR – the problems are still an issue, and are still preventing me from getting results consistent enough to feel comfortable publishing.
Copy that and … THANKS
Copy that and … THANKS AGAIN!
Re:
https://www.pcper.com/ima
Re:
https://www.pcper.com/image/view/89768?return=node%2F69337
To clarify, I am still curious if the latter AIC will be bootable when installed in an AMD Threadripper motherboard.
For prosumers and workstation users who now prefer a TR system, this AIC with 4 x Optanes could work.
Nevertheless, you are correct: 4 x Samsung 960 Pro make more sense in such an AIC, for reasons you have already explained in your other reviews.
Yes, it’s bootable since the
Yes, it's bootable since the UEFI initializes the array each boot. Consistent performance? that's another story entirely.
Apples-to-apples, perhaps we
Apples-to-apples, perhaps we should wait for M.2 Optanes to be enhanced with support for x4 PCIe lanes each.
Also, there is the 16 GHz clock rate approved for PCIe 4.0:
Intel may be “vectoring” x2 PCIe lanes to a future date
when x2 @ 16G = x4 @ 8G .
https://www.anandtech.com/show/11967/pcisig-finalizes-and-releasees-pcie-40-spec
PCIe 4.0 is gopping to be
PCIe 4.0 is gopping to be power hungry at least initially. Don't be surprised to see SSDs hang out at 3.0 for a good while after 4.0 is available and shipping on motherboards.
So I just want to make sure I
So I just want to make sure I can use this….I have an Asus Z370E, 8700K, one 1080Ti….with 4 hard drives (3 standard- 2@7200RPM, one 5400RPM) and one SSD boot drive from Crucial. Also, External backup HD from western digital. Will I have any issues if I decide to purchase this?
From what I been told the
From what I been told the Intel Optane SSD 800P has to be boot drive.
It doesn’t *have* to be a
It doesn't *have* to be a boot drive. Can be used as a fast random access temp drive, etc. All depends on where you want the traits of this particular SSD.
*edit* you may be thinking of Optane Memory (caching), which will only cache the boot drive. 800P can be used this way, by installing the Optane Memory driver, but it's overkill with >32GB of cache.
The Z370E has a pair of M.2
The Z370E has a pair of M.2 slots, so even if your Crucial SSD is M.2, you should have room. You could put the 800P in the primary M.2 slot to use for boot and shift any other (larger) M.2 SSD to the second M.2 / other SATA port (depending on what it is) for use as a game / other SSD.
Interesting graph:
history
Interesting graph:
history of PCIe actual bandwidth
compared to
every 3 years, I/O bandwidth doubles:
https://images.anandtech.com/doci/11967/pci-sig_history_graphic_wide_rgb_0533_575px.jpg
Allyn, On behalf of all
Allyn, On behalf of all pcper.com users,
please accept our sincere appreciation
for your very prompt and professional
answers to all our questions.
I just got great deal on a i7
I just got great deal on a i7 7800x and a Gigabyte Aorus Gaming Ultimate,
can I use say a 960 Pro 1 Tb as boot then the 2 118gb optane as a raid cache.
I won’t be using high end gpu this will be a sever build with alot of Hdds?
Any news on Micron’s QuantX ?
Any news on Micron’s QuantX ?
From their earnings call I
From their earnings call I don’t think we’ll see it until 2019
Pretty sure that was just
Pretty sure that was just vaporware to force intel to release hypetane asap. as there has been zero products from micron with there brand of xpoint.
i was excited to see the dimms they were working on it all looked so good on paper but sadly that will most likely be a pipe dream that will never bear fruit.
you got about as much chance as valve releasing HL3 then micron actually make a product with xpoint at this point.
Hi Allyn, i’m just curious about “Intel’s recommended client SSD conditioning pass”,could you tell me where can i find those recommendation?
I reviewed and nothing there.
Allyn’s not here man … he’s writing those things now!
I think this is the paper he was referring to https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ssd-server-storage-applications-paper.pdf
Thank you Jeremy!
I think i misunderstood his thoughts, and now i’m clear that leaving 8GB portion for random access is a better way to simulate real-world condition.