Last month in Tobii's suite at CES I was given a demonstration of a prototype VR headset that looked like any other HTC Vive – except for the ring of Tobii eye-tracking sensors inside and around each lens. While this might seem like a bit of an odd concept at first I was patient as the benefits were explained to me, and then blown away when I actually tried it myself.
As you know, if you have used a VR headset like the Oculus Rift or HTC Vive, the basic mechanics of VR interaction involve pointing your head in the direction you want to look, reaching with your hand (and controller) to point to an object, and then pressing a button on the controller to act. I will be completely honest here: I don't like it. After a little while the fatigue and general unnatural feeling of rapid, bird-like head movements kills whatever enthusiasm I might have for the experience, and I was the last person to give high praise to a new VR product. HOWEVER, I will attempt to explain why simply adding eye tracking actually made the entire experience 1000 times better (for me, anyway).
When I put on the prototype headset, the only setup I had to do was quickly follow a dot in my field of vision as it moved up/down/left/right, like a vision test for a driver's license. That's the entire calibration process, and with that out of the way I was suddenly able to look around without moving my head, which made the head movements when they followed feel completely natural. I would instinctively look up, or to the side, with my head following when I decided to focus attention on that area. The amount of physical head movements was reduced to normal, human levels, which alone prevented me from feeling sick after a few minutes. Of course, this was not the only demonstrated feature of the integrated eye-tracking, and if you are familiar with Tobii you will know what's next.
This looks primitive, but it was an effective demo of the eye-tracking integration
The ability of the headset to know exactly where you are looking allows you to aim based on your line of sight if the game implements it, and I tried some target practice (throwing rocks at glass bottles in the demo world) and it felt completely natural. After launching a few rocks at distant bottles I instantly decided that this should be the mechanic of fantastic VR football video game – that I could throw at different receivers just by looking them down.
I also received a demo of simulated AR integration (still within the VR world), and a demo of what eye-tracking adds to a home theater experience – and it was pretty convincing. I could scroll around and select movie titles from an interface by simply looking around, and within the VR world it was as if I was looking up at a big projection screen. Throughout the different demos I kept thinking about how much more natural everything felt when I wasn't constantly moving my head around and pointing at things with my controller.
Finally, there was another side to everything I experienced – and it might have been the most interesting thing from a PC enthusiast perspective: if the VR headset can track your focus, the GPU doesn't have to render anything else at full resolution. That alone could make this something of a breakthrough addition to the current VR headset space, as performance is very expensive (even before the mining craze) and absolutely necessary for a smooth, high frame-rate experience. After 45 minutes with the headset on, I felt totally fine – and that was a change.
So what is the takeaway from all this? I'm just an editor who had a meeting with Tobii at CES, and I walked out of the meeting with a couple of business cards and nothing else. I admit that I am a VR skeptic who went into the meeting with no expectations. And I still left thinking it was the best product I saw at the show.
More information and media about the CES demos are available from Tobii on their CES blog post.
great write up sebastian
how
great write up sebastian
how long before we see this tech in headsets for sale?
if it changed your thinking on VR, it must be huge
Thanks! And I tried to get
Thanks! And I tried to get that information – but nothing doing. There haven't been any products announced so far, but I have to imagine we'll hear something this year considering the prototype I was wearing seemed like it was ready for production.
thanks so much
hoping sooner
thanks so much
hoping sooner than later of course
“except for the ring of Tobii
“except for the ring of Tobii eye-tracking sensors inside and around each lens. ”
Not sensors, those are nIR LED illuminators. The sensor will be a camera hidden inside the HMD.
Because the illuminators are off-axis, they will be using Dark Pupil tracking. Because they are using multiple illuminators, they are likely just using regular pupil tracking rather than augmented tracking using the Purkinje reflections. It is possible they are sequentially pulsing the illuminators to measure the Purkinje reflections from multiple axes though, which could allow some degree of 3D pupil tracking, rather than regular 2D tracking + sphere model as with Dark Pupil tracking alone. 2D tracking allows for a good estimate of gaze target, but doesn’t tell you much about actual pupil position, and demands calibration before every use for accurate gaze targets. 3D pupil tracking (6DoF), does not require calibration (because you know true pupil position so do not need to calibrate for offset) and has the added benefit of allowing you to use pupil pose for lens distortion compensation as the pupil moves, rather than the current state-of-the-art of a fixed lens distortion function that is ‘good enough’ for most pupil poses rather than optimised for true pose.