Measuring VR Performance and Experiences – A Prologue

More Complications – Matching Experienced Performance

It’s not quite that simple though, there are more complications involving the experience of using a VR headset compared to the performance data we showed you on the previous page. The most interesting example we can show revolves around the ability for OpenVR to warp displayed frames when an 11.1ms frame time is missed. The idea behind frame warp is amazing: if you don’t have a NEW frame to show when you come to a 90Hz refresh cycle on the HMD, use the previous frame and shift it slightly (based on speed and location data) in the direction of head movement to simulate where the next frame should be.

This has a couple of interesting side effects.

1. If you look at the right place at the right time, you can see an effect called “pull in” where the side of the display shows encroaching blackness as the warp moves the frame on the display without new game data to show.
    
2. The user can effectively experience two frame rates – the refresh rate of the screen with warping to smooth out rotation and the frame rate of the rendered objects in the game scene itself.

Let’s focus on the second item above.

When we use The Gallery on a GTX 970 with the High image quality preset, we can force the game to run at a 45 FPS render rate. If you do something simple like take your avatar hands and move them back and forth in front of your face while looking at the fire, you will see a 45 FPS animation. For those of us that are used to looking for these kinds of image issues, it’s pretty obvious what we were seeing. If you make the same hand motions in front of your face, but look up at the sky instead, where the game is reporting a 90 FPS performance rate, the animation is significantly smoother.

However, if you are in the 45 FPS scenario, and waving your hands in front of your view, if you also rotate your head on a static axis to the left or right, THAT movement will appear to be running at a higher frame rate. That is due to the warp capability of the OpenVR (and Oculus VR) system to slightly move the entire image on the headset with a 90Hz interval frame is missed. The result is you’ll see what appears to be 90 FPS motion to the left or right (or up or down) while the hands on the screen will be running at 45 FPS effectively.

We created video to show you this phenomenon. Using 960 FPS capture on a Sony camera, we have slowed down the animation to demonstrate the effect we are seeing. What you should look for is that with the GTX 970, the hand rotation animation will update a slower rate than the rest of the environment you can see will update, on order of 1:2.

We have zoomed in and slowed things down here to help show you the effect, and removed 9 frames of black between each frame of animation (these screens have AMAZINGLY low pixel persistence). I think the result speaks for itself. Obviously the idea of artificially creating this scenario with hands waving erratically in the air is artificial, but you would experience this same effect in any situation where the frame rate of the game is running at 45 FPS and you are interacting with the world rather than just looking around.

As a real world example, imagine a huge, 100-in monitor running on your desk, playing a game at 45 FPS. If you move your avatar's hands in the game, you'll see a 45 FPS animation. But if you move your physical, real-world head from side to side while looking at your monitor, THAT movement will be much smoother. That's the effect we are describing. 

How is it possible to get the effect of a 90 FPS rotation while the game is running at 45 FPS? Warp. Timewarp is the ability for the SteamVR system to move the frame on the HMD slightly without input from the game if a new frame has not reached the display at the specified 90Hz / 11.1ms time.

And we can show it in action. Again, using a fast frame rate capture, this time a 240 FPS GoPro, we were able to see timewarp working as we spin the HTC Vive to the right on the GTX 970 test system in a 45 FPS environment (that trusty campfire). (Note this video looks more flickery because of the GoPro frame rate.)

If you watch closely you can see black “pull in” effect of timewarp – because there are no new pixels from the game engine to show in that location as the frame is moved with the direction of head movement. As the game catches up that pull in effect is corrected. As we are in static 45 FPS location for testing, that pattern repeats.

This is just the beginning…

The technologies that goes into building and providing a world class VR experience with the HTC Vive and SteamVR are astounding. The more we dive into the hardware and the software, the more impressed I become. But that doesn’t mean we aren’t going to be pushing all parties involved to improve things through performance evaluation and experiential testing.

I’d like to thank Cloudhead games for working with me on this project with The Gallery. But it is important for readers and other media out there to recognize that not all games will be able to be profiled in this manner. The configuration of the companion viewport seems to vary from game to game and from platform to platform. And though we have the SteamVR performance heuristic tool set to play around with, Oculus has not provided anything like it (that I am aware of).

We still have a lot of work to do as a community. Hardware vendors, software engineers, analysts, consumers – all of us. I’m looking forward to the on-going discussion and to putting in many more weeks of work to find the best way to analyze and measure VR performance.