PCMark Vantage and 7

For trace based testing, we are replacing the outdated PCMark 05 results with a pair of newer versions of the same. PCMark Vantage and PCMark 7 results will be included moving forward. We are also running PCMark 8 on samples as we receive them, however we do not yet have sufficient comparative data to make a useful chart. Vantage results tend to be higher as they show more of a raw / peak type of result measured during the trace, while 7 shows more of an averaged value over the time duration of the trace. Translation: Vantage shows max throughput while 7 shows how that throughput impacts overall system performance. You can easily tell the difference by looking at the results of the HDD (VelociRaptor), which scores higher under 7 since the overall system is usually doing other things in addition to accessing storage.

We are considering discontinuation of the PCMark suite in our testing. We have a spread of models included here, which vary legitimately and greatly in metrics for other tests, yet PCMark places them mostly dead even with each other. Further, the test that shows the greatest variance among samples ('Application loading' in Vantage / 'Starting applications' in 7) is nearly an inversion when both sets of results are compared against each other. The SP610 is a clear winner in Vantage but at the bottom of the pack in 7.

Spot checks we've performed with PCMark 8 don't show any improvement in this area – results from a mid-grade SATA SSD were within 1-2% of the results from a recent PCIe SSD capable of 2GB/sec. It appears PCMark is placing far too much weight on things other than storage during their storage-only test. PCMark 8 does have an Expanded Storage Test, but I don't feel its workload pattern is anywhere near what I would consider an actual performance over time metric. Your thoughts are welcome in the comments below.

« PreviousNext »