Tool Validation – Hardware vs Software Capture
In the description above, part of the development process of FCAT VR was the need to validate any metrics it would provide in a software-only environment with both experiential and hardware-based capture data points. Playing a game in the headset and correlating with performance data is a substantial part of making sure FCAT VR works, but more critical to confirmation is the hardware-based video capture of the VR headset input, with appropriate placed indicator overlays, to post-process analyze the video and allow us to clearly see effects of features and performance.
Running both a hardware and a software capture on a VR game at the same time is very possible but requires the use of multiple keyboards or multiple people. Perfectly matching up the data sets requires the use of the FCAT VR Analyzer software’s ability to shift and crop data, giving us the best chance to correlate the collections.
Click to Enlarge
In the above graph, the green line is the hardware-based capture and the green line is the software-based capture. First notice that the green line only shows frame times at either 11ms, 22ms or 33ms (slightly rounding). Because the images going to the headset always have vertical sync enabled, we do not get instantaneous frame rates or frame time that vary from this pattern. The blue line looks more like a frametime graph we utilize during performance evaluation of hardware on PC Perspective and it is calculated using the data from Oculus and SteamVR APIs. It adds up the relevant render times, including CPU and GPU workloads, and presents that information for each frame.
After doing some slight adjustment to the placement of the data, the two data sets match up. When we see spikes in the hardware captured data we almost always see corresponding spikes in the software captured data. Delivered frame rates of 45 and 90 FPS can be identified when the frame time exceeds some value at or near 11ms. Look at the transition that occurs between 8s-10s on the time line: the frame time is hovering around 16ms and then decreases to 10ms, and when it crosses a threshold (which will be slightly different for every game) you see the hardware captured frametime drop from 22ms (45 FPS) to 11ms (90 FPS). Other than a few spikes during testing, the GTX 1060 running Obduction can maintain a 90 FPS mark through rising frametimes until we hit the 30s mark when it jumps back to 45 FPS.
Click to Enlarge
When we add in the interval plot graphs things get a bit more crowded, but also more interesting. Remember that the interval plots are a representation of the experienced frame rate provided by the game through the platform hardware. It attempts to measure new frames, dropped frames and “synthetic” frames that are generated by the software (of which only Oculus currently performs). Clearly, we see that the game beings at the 45 FPS mark, increases to 90 FPS at 9s, and then drops back to 45 FPS at 30s. While the upper interval plot shows these sections in red, the lower interval plot does so in mostly yellow. Why? The hardware capture (top) is unable to discern the difference between a dropped frame that has been reprojected and a frame that has been synthesized/created by the Oculus run time. Because of that it assumes all are drops. The software capture plot (bottom) has the knowledge of this created frame from the Oculus API and can note the difference between them. There can be (and still are) dropped frames in the software capture results.
Click to Enlarge
The FCAT VR Analyzer tool allows us to zoom in on any section of the data as well for a closer look. In this section covering about 10 seconds, it allows us to see that the software captured data shows a slight step pattern down from 45 FPS to 90 FPS and a dropped frame directly before the transition occurs (which we see happens very consistently in our testing with the Rift). The interval plots ability to express the perceived experience of the VR game at any point is valuable and helps us compare different data and hardware.
Let’s have a look at another data collection, this time from the Radeon RX 480 in Chronos (Rift).
Click to Enlarge
In this graph the red line is the RX 480 in hardware capture while the purple line is the RX 480 in software capture. Again, after some manual alignment in the analyzer tool, you can see the matching of spikes and frame rates. The results also show a clear threshold when the frame rate jumps from 90 FPS to 45 FPS and back. Both data sets, and numerous others we have generated over the last several months with the tool, continue to emphasize the accuracy of reporting that FCAT VR provides for both primary GPU hardware vendors.
Click to Enlarge
In the above screen shot I have added the interval plots for both RX 480 data sets; again they match up as expected with the red/yellow color difference existing because of the ability for the on-system software version to understand when a synthesized frame has been created.
Click to Enlarge
Finally, another zoomed in view to get added detail from the frametime and interval plots.
Great write-up! Your
Great write-up! Your GeForce.com link for FCATVR in the last sentence of the article is pointing to a file location on your C drive instead of Nvidia’s site though. This tool looks really interesting and I look forward to playing with it at some point myself 🙂
What, you can access my hard
What, you can access my hard drive? Huh… 😉
Thanks!
Better than it was, but now
Better than it was, but now it takes me to the first page of the article instead of the download page on Nvidia’s site.
Fixed… Again!
Fixed… Again!
“Gentlemen, we can correct
“Gentlemen, we can correct this link; we have the technology…” 😉
Thanks Ken!
Thanks for all your hard work
Thanks for all your hard work guys!
Tests I’d like to see:
Higher
Tests I’d like to see:
Higher performing GPUs (Fury, 1070)
SLI/CF
7700K-based build (better for games than 5960X)
Interesting that the RX 480
Interesting that the RX 480 loses in all Unreal Engine 4 games (and Dirt Rally, don’t know about Codemasters Engine), an engine that is heavily tailored to Nvidia cards. Look at the only game using Unity, Edge Of Nowhere and, surprise, surprise, the RX 480 magically catches up to the GTX 1060. *rolleyes*
I wish more VR devs would use Unity or the UE devs build in better support for AMD Hardware.
It’s clear that amd is in
It’s clear that amd is in just stay alive mode in the gpu space, similar to where they were against Intel for the last four years with Bulldozer. I really don’t expect Vega to be competitive for more than a month or two as Volta is right around the corner. It’s a bad time to a gpu enthusiast from a competition perspective.
You could ignore AMD and just
You could ignore AMD and just buy nVidia for GPU and Intel for CPU like any other normal person. You’re not poor are you?
By definition, ‘normal’
By definition, ‘normal’ people are not ‘rich’, as they are considered ‘average’ in wealth, hence the concept of the ‘rich’, ie. above average in wealth, having more money than the average person.
Just a tip. You’re not going to win any friends on here spouting elitist cr@p like that. Nobody here cares how much money you think you have or actually have. It’s utterly irrelevant. What matters is whether you are a person of substance, rather than a shallow, materialistic looser.
How can the delivered fps be
How can the delivered fps be higher than the unconstrained fps? See the “Dirt Rally – High Preset” result on the “GTX 1060 vs RX 480” page
Check the last paragraph on
Check the last paragraph on the “GPU Performance Scaling” page. It explains that phenomenon.
Thx for pointing that out,
Thx for pointing that out, here’s the quote for the even lazier than me:
“Because the frametimes reported by the Oculus and SteamVR runtimes combine the CPU and GPU, but the unconstrained time is based on GPU alone. In situations where the frametime is very close to the 11ms/90FPS mark we see overlaps of CPU and GPU time (rather than it being purely sequential). The result is that unconstrained FPS will sometimes be lower than delivered FPS.”
However, I still don’t get it, I’m sorry. Perhaps a few more words are in order?
My interpretation of this is
My interpretation of this is that in this scenario, the CPU is getting some work done while the GPU is working; aka the system is performing better/faster simultaneously instead of work having to wait for one to finish before moving to the next compute device for the current operation.
Unconstrained FPS doesn’t take the extra performance-gain into account, so when only looking at the GPU and expectation of frame delivery, it’s presenting a slower/incomplete analysis of performance. It doesn’t pick up that the system is doing some of the work at the same time instead of sequentially, so it’s analysis is that it’s taking longer than reality shows.
The internet will correct me if I’m wrong, but that was my interpretation of the situation.
So, Nvidia wants us to wait
So, Nvidia wants us to wait for AMD Vega for VR? OK, I’ll wait.
What is the shader:ROP ratios
What is the shader:ROP ratios for both the GTX 1060 and the RX 480? And it again looks like time will tell as to what if any gaming software and driver/API tweaks will do for AMD’s polaris based SKUs! The RX 480 and Polaris SKUs are getting some tweaks in the form of the RX 580/other Polaris updates so maybe some retesting should happen then.
There is also maybe some driver/games/other software improvments coming along with any of the Polaris/Refresh RX 500 series SKUs so this very same bemchmark run on the RX 500 series SKUs and maybe some feature support that the RX 480 lacks compared to the GTX 1060.
I also want to see any current CF/SLI VR benchmarking along with any DX12/Vulkan explicit multi-GPU adaptor benchmarking where both CF/SLI are not used and any Dual GPU usage is done through DX12’s/Vulkan”s API multi-GPU load balancing methods.
Vega is still a little ways off so some good looks at the RX 500 series Polaris refresh SKUs like any RX 480/RX 580 to GTX 1060 benchmarking, with some added multi-GPU VR/Non VR setups are in order. The RX 480’s are dropping in price below the $200 dollar price point and if the RX 580s retail at around $199 then the RX 480 pricing will most likely hit the 150 price point at some point in time on any stock of RX 480s that may be remaining in the retail channels! And that bodes well for some affordable RX 480 CF gaming uaseg for VR/Non VR gaming usage with the new Ryzen/AM4 options that are now available and without.
The Rx 480 is $179, correct ?
The Rx 480 is $179, correct ? Isn’t the GTX 1060 40% more ?
They seem to be different class of GPU based on price.
The Radeon Fury seem to be the direct competitor to the 1060, no ?
Exact same price as the 1060.
Would be cool when comparing cards, that the same class or card (based on price) get compared.
Because I’m pretty sure the GTX 1080 ti crush the 1060 at VR…
but then its also pointless the compare them both when they dont cost the same.
depends where you live. in my
depends where you live. in my country the two still pretty much about the same price. sometimes a bit cheaper than 1060. also RX480 with such price still depending on deal. AFAIK AMD has not officially cut RX400 series price.
the thing about geforce is they tend to sell well even without price cut so retailer have no reason to offer big deal on them unlike AMD cards that is much harder to sell.
“the thing about geforce is
“the thing about geforce is they tend to sell well even without price cut so retailer have no reason to offer big deal on them unlike AMD cards that is much harder to sell.”
Sigh. Yes, the legions of the stupid strike again. I’m going to really enjoy watching AMD do to nVidia what they just did to Intel. 🙂
Damn the GTX 1060 is stompin
Damn the GTX 1060 is stompin RX 480. Really want to get into performance testing myself.
Curious what model of the rx
Curious what model of the rx 480 and 1060 was used?
The gap in unconstrained FPS make little sense,
unless the rx480 may have been running at like 1ghz ?
I also found anything below 90fps in VR gives a very poor experience. its not like a game on a monitor going from 60 to 40fps, its noticeable but ok.. in VR dropping below 90fps is headache inducing.
So here the “hardocp” method would be good to use.
in short. what is the game setting to use to reach near rock solid 90fps.
can the gtx 1060 even reach that, and be used for solid VR ?
From this it seem it falls short with the tested settings.
We’re using the standard
We're using the standard RX480 and GTX 1060.
Very interesting paper, but
Very interesting paper, but you should give more technical data on the videocards you tested. parameters like memory or GPU frequency, Oced or not, driver version, etc. are important. Not all 1080, 1060, 480 are born equal.
Standard cards were used, and
Standard cards were used, and drivers were current as of time of testing for the review.