PC Perspective Custom Test Suite Introduction
A New Test Suite
We have implemented radically new test methodology. I'd grown tired of making excuses for benchmarks not meshing well with some SSD controllers, and that matter was amplified significantly by recent SLC+TLC hybrid SSDs that can be very picky about their workloads and how they are applied. The complexity of these caching methods has effectively flipped the SSD testing ecosystem on its head. The vast majority of benchmarking software and test methodologies out there were developed based on non-hybrid SLC, MLC, or TLC SSDs. All of those types were very consistent once a given workload was applied to them for long enough to reach a steady state condition. Once an SSD was properly prepared for testing, it would give you the same results all day long. No so for these new hybrids. The dynamic nature of the various caching mechanisms at play wreak havoc on modern tests. Even trace playback testing such as PCMark falter, as the playback of traces is typically done with idle gaps truncated to a smaller figure in the interest of accelerating the test. Caching SSDs rely on those same idle time gaps to flush their cache to higher capacity areas of their NAND. This mix up has resulted in products like the Intel SSD 600p, which bombed nearly all of the ‘legacy’ benchmarks yet did just fine once tested with a more realistic, spaced out workload.
To solve this, I needed a way to issue IO's to the SSD the same way that real-world scenarios do, and it needed to be in such a way that did not saturate the cache of hybrid SSDs. The answer, as it turned out, was staring me in the face.
Latency Percentile made its debut a year ago (ironically, with the 950 PRO review), and those results have proven to be a gold mine that continues to yield nuggets as we mine the data even further. Weighing the results allowed us to better visualize and demonstrate stutter performance even when those stutters were small enough to be lost in more common tests that employ 1-second averages. Merged with a steady pacing of the IO stream, it can provide true Quality of Service comparisons between competing enterprise SSDs, as well as high-resolution industry-standard QoS of saturated workloads. Sub-second IO burst throughput rates of simultaneous mixed workloads can be determined by additional number crunching. It is this last part that is the key to the new test methodology.
The primary goal of this new test suite is to get the most accurate sampling of real-world SSD performance possible. This meant evaluating across more dimensions than any modern benchmark is capable of. Several thousand sample points are obtained, spanning various read/write mixes, queue depths, and even varying amounts of additional data stored on the SSD. To better quantify real-world performance of SSDs employing an SLC cache, many of the samples are obtained with a new method of intermittently bursting IO requests. Each of those thousands of samples is accompanied by per-IO latency distribution data, and a Latency Percentile is calculated (for those counting, we’re up to millions of data points now). The Latency Percentiles are in turn used to derive the true instantaneous throughput and/or IOPS for each respective data point. The bursts are repeated multiple times per sample, but each completes in less than a second, so even the per-second logging employed by some of the finer review sites out there just won’t cut it.
Would you like some data with your data? Believe it or not, this is a portion of an intermittent calculation step – the Latency Percentile data has already been significantly reduced by this stage.
Each of the many additional dimensions of data obtained by the suite is tempered by a weighting system. Analyzing trace captures of live systems revealed *very* low Queue Depth (QD) under even the most demanding power-user scenarios, which means some of these more realistic values are not going to turn in the same high queue depth ‘max’ figures seen in saturation testing. I’ve looked all over, and nothing outside of benchmarks maxes out the queue. Ever. The vast majority of applications never exceed QD=1, and most are not even capable of multi-threaded disk IO. Games typically allocate a single thread for background level loads. For the vast majority of scenarios, the only way to exceed QD=1 is to have multiple applications hitting the disk at the same time, but even then it is less likely that those multiple processes will be completely saturating a read or write thread simultaneously, meaning the SSD is *still* not exceeding QD=1 most of the time. I pushed a slower SATA SSD relatively hard, launching multiple apps simultaneously, trying downloads while launching large games, etc. IO trace captures performed during these operations revealed >98% of all disk IO falling within QD=4, with the majority at QD=1. Results from the new suite will contain a section showing a simple set of results that should very closely match the true real-world performance of the tested devices.
While the above pertains to random accesses, bulk file copies are a different story. To increase throughput, file copy routines typically employ some form of threaded buffering, but it’s not the type of buffering that you might think. I’ve observed copy operations running at QD=8 or in some cases QD=16 to a slower destination drive. The catch is that instead of running at a constant 8 or 16 simultaneous IO’s as you would see with a saturation benchmark, the operations repeatedly fill and empty the queue, meaning the queue is filled, allowed to empty, and only then filled again. This is not the same as a saturation benchmark, which would constantly add requests to meet the maximum specified depth. The resulting speeds are therefore not what you would see at QD=8, but actually, a mixture of all of the queue steps from one to eight.
Conditioning
Some manufacturers achieve unrealistic ‘max IOPS’ figures by running tests that place a small file on an otherwise empty drive, essentially testing in what is referred to fresh out of box (FOB) condition. This is entirely unrealistic, as even the relatively small number of files placed during an OS install is enough to drop performance considerably from the high figures seen with a FOB test.
On the flip side, when it comes to 4KB random tests, I disagree with tests that apply a random workload across the full span of the SSD. This is an enterprise-only workload that will never be seen in any sort of realistic client scenario. Even the heaviest power users are not going to hit every square inch of an SSD with random writes, and if they are, they should be investing in a datacenter SSD that is purpose built for such a workload.
Calculation step showing full sweep of data taken at multiple amounts of fill.
So what’s the fairest preconditioning and testing scenario? I’ve spent the past several months working on that, and the conclusion I came to ended up matching Intel’s recommended client SSD conditioning pass, which is to completely fill the SSD sequentially, with the exception of an 8GB portion of the SSD meant solely for random access conditioning and tests. I add a bit of realism here by leaving ~16GB of space unallocated (even those with a full SSD will have *some* free space, after all). The randomly conditioned section only ever sees random, and the sequential section only ever sees sequential. This parallels the majority of real-world access. Registry hives, file tables, and other such areas typically see small random writes and small random reads. It’s fair to say that a given OS install ends up with ~8GB of such data. There are corner cases where files were randomly written and later sequentially read. Bittorrent is one example, but since those files are only laid down randomly on their first pass, background garbage collection should clean those up so that read performance will gradually shift towards sequential over time. Further, those writes are not as random as the more difficult workloads selected for our testing. I don't just fill the whole thing up right away though – I pause a few times along the way and resample *everything*, as you can see above.
Comparison of Saturated vs. Burst workloads applied to the Intel 600p. Note the write speeds match the rated speed of 560 MB/s when employing the Burst workload.
SSDs employing relatively slower TLC flash coupled with a faster SLC cache present problems for testing. Prolonged saturation tests that attempt to push the drive at full speeds for more than a few seconds will quickly fill the cache and result in some odd behavior depending on the cache implementation. Some SSDs pass all writes directly to the SLC even if that cache is full, resulting in a stuttery game of musical chairs as the controller scrambles, flushing SLC to TLC while still trying to accept additional writes from the host system. More refined implementations can put the cache on hold once full and simply shift incoming writes directly to the TLC. Some more complicated methods throw all of that away and dynamically change the modes of empty flash blocks or pages to whichever mode they deem appropriate. This method looks good on paper, but we’ve frequently seen it falter under heavier writes, where SLC areas must be cleared so those blocks can be flipped over to the higher capacity (yet slower) TLC mode. The new suite and Burst workloads give these SSDs adequate idle time to empty their cache, just as they would have in a typical system.
Apologies for the wall of text. Now onto the show!
Finally, it crazy how long
Finally, it crazy how long it’s taken to get a reasonable competitor to the Samsung NVME juggernaut! At least it’s competitive price and performance wise with the 960evo.
This is a very interesting
This is a very interesting NVMe M.2 drive but the 960 evo is barely any more expensive at this point. 10% cheaper isn’t going to make up for the large performance delta.
960 EVO offers only 3 year
960 EVO offers only 3 year warranty which is quite a difference. Yet I will not buy a single intel product anymore unless the performance delta favours them immensely, good bye asshole corp.
That’s why it didn’t get
That's why it didn't get Editor's Choice. It would need to have outperformed the 960 in more ways than it did for me to go that far in the recommendation. If the price delta is $10-20, I'd personally still buy the EVO today. Still a good showing from Intel through – the 960's needed some healthy competition.
“I’m awarding gold to the
“I’m awarding gold to the 256GB and 512GB models of the 760p. These products nearly match the current M.2 NVMe class leader, and win in some of our more critical metrics, all while coming in at a lower cost.”
Totally corrupt /s
Dude go get your tinfoil hat and play in the corner.
A white paper doesn’t lie about a product, it put the strengths on display and show when it would make sense to choose one product over another. Allyn is one of the best storage editors out there, of course they would go to them to write a third party paper. You wouldn’t go to LTT for this kind of in depth reporting, they aren’t geared for that type of work. Also why duplicate work or not use work you gained in the research of a product in your own sites review?
You seemingly don’t
You seemingly don’t understand how conflict of interest pertains to journalism. A conflict of interest exists regardless of whether this conflict ends up influencing Allyn’s review at PCPer. Ultimately, it is the responsibility of any proper journalist to keep a professional distance (read: financial independence) from the subject of their coverage.
This has nothing to do with whether Allyn should have been chosen over some other youtube reviewer (hint: no reviewer should conduct paid work for a vendor whose products they review). If you are a journalist/reviewer, you have the responsibility to ensure that you are not in any position where you stand to personally benefit from your professional conduct. It is absolutely unacceptable to be paid by a company (for real work), and fail to disclose this financial relationship to your readers.
This is such a blatant example of COI that I’m shocked they thought it would go unnoticed. To answer your question: if you were paid by a company (Intel) to perform work for them, you stand to benefit from them continuing to pay you, or provide you with other benefits (like privileged access to products, or early access). Adored’s video discussed how PCPer’s access to optane did not reflect the relative size and reach of their outfit (read: they were given privileged access to hardware that was not available to the rest of the press). This (indirectly) has monetary value, since it allowed PCper to produce content that other outlets could not feasibly produce. Unique content results in views, and therefore money. Readers have the right to know that this relationship existed, and PCPer knowingly chose not to disclose any such relationship. It’s extremely disappointing, and this is coming from a frequent consumer of PCPer content.
To be clear, we duplicate the
To be clear, we duplicate the work regardless. It would be extremely unlikely for any possible white paper work / other research work to use an identical test configuration as the test suite used for reviews, and even if it were, I'd do separate work for both sides anyway.
Shrout Research’s commercial
Shrout Research’s commercial conflict of interest makes this site in best case questionable. Sorry Allyn and Ryan, your credibility is in the gutter for now. 🙁
PCPer is now dead to me. In
PCPer is now dead to me. In nearly 35 years of IT work I have never seen such a serious conflict of interest as this one. Everything that now comes out of PCPer’s so-called journalists mouths will be nothing but meaningless blablabla to me. PCPer needs to be served with a Class Action Lawsuit, at the very least.
The only surprise is that the
The only surprise is that the AMD fanboy community still watches AdoredTV after all his BS from the previous two years. You guys are seriously in love with siege mentality.
Error with results 256gb:
1.
Error with results 256gb:
1. Saturated vs. Burst Performance (for 128 gb (two graphic)).
It doesn’t appear there is
It doesn’t appear there is any spare area on these drives. Would it be worthwhile to overprovision them to say 250GB, 500GB etc ?