Bringing foveated rendering and analytics to the BMW M Virtual Experience

The VIVE Pro Eye is the latest in HTC VIVE's line of high-end VR head-mounted displays (HMDs). Designed with integrated eye tracking, it offers a number of significant opportunities for enterprise-level experiences. As announcement partners for the headset, ZeroLight combined the new hardware with the latest NVIDIA GPUs to elevate one of its most popular VR projects, the BMW M Virtual Experience, and support the HMD's launch at CES 2019.

"This collaboration with ZeroLight embodies a powerful shift in modern automotive retail. Combining high-end visualisation with detailed behavioural insights, digital experiences such as these give brands the power to personalise every interaction and encourage upsells."

Anita Tonini, Chief Financial Officer, BMW Group Central and Southeastern Europe

The original experience, created to enhance the exclusive 2018 M Drive Tour event, invited users to step onto a simulated track at Le Mans to configure and explore the BMW M5 in virtual reality. The experience was first shown trackside at the BMW event itself before being showcased at London Tech Week 2018.

The integration of eye-tracking technology in the new HMD enabled ZeroLight to add two major advancements to the experience: user behaviour insights and foveated rendering.

‍

"Now with improved visual quality and detailed customer behaviour insights, the BMW M Virtual Experience is a powerful example of automotive marketing's technologically advanced future. We are proud to see the impact this project's continued progression has had both within our industry and beyond."

_{Florian Stiller, Head of Event and Sportmarketing BMW Group Central and Southeastern Europe.}

Foveated rendering
‍Foveated rendering is a technique that increases visual quality while reducing the rendering workload. This is made possible because the human eye only perceives the area at which the person is looking (the foveal area) in high resolution; the detail of everything in the peripheral area is significantly lower. Foveated rendering mirrors this functionality of the eye to increase visual quality in the foveal while reducing the resolution of the remainder of the field of view (FOV).
‍
Because foveated rendering replicates natural vision in this way, users simply notice a sharper, more immersive image and have no idea that the resolution outside of the foveal is reduced. This presents a significant opportunity for providers of high-end VR solutions where visual fidelity is key, as they can increase the quality of users' experiences without the additional expense of more powerful hardware.

"Foveated rendering is a huge advancement for VR. We're proud to be leading it. With VIVE Pro Eye's integrated eye tracking, developers can maximize processing performance into what users see in the moment. It leads to better visual fidelity and a better experience for customers. It's where the VR industry is heading—BMW and ZeroLight are already there."

_{JB McRee, Director of Marketing - Enterprise Virtual Reality, HTC VIVE}

For this project, ZeroLight used NVIDIA's latest Turing GPU, the Quadro RTX 6000, to develop an optimal foveated rendering technique that avoids any additional CPU overheads by taking advantage of Variable Rate Shading (VRS). The video below demonstrates the eye tracking and foveated rendering functionality in action: the green area is the high-resolution foveal; the yellow is the blend area; the red is the remainder of the FOV.

‍

‍

ZeroLight was able to use the performance benefits brought about by this technique to render at 9x the resolution of the VIVE Pro Eye while maintaining a 90fps refresh rate. This was achieved using a technique called supersampling, which renders at a higher resolution and then filters the data to ensure a detailed and smooth image without any aliasing. This creates an exceptionally immersive experience. More detail on the impact of foveated rendering can be found in the project's tech development blog.

User behaviour insights

Mapping out specifically where a user looks during an experience can also give useful insights into behaviour and preferences. ZeroLight used this information to enhance its Wisdom VR platform, which can track anonymous interactions 10 times every second, in line with GDPR.

With this wealth of user information, retailers will be able to see specifically where their customers are looking in VR. This information can be analysed on an individual or aggregated basis, enabling sales teams to make a range of informed decisions about preferences.

"This project demonstrates how modern advancements can significantly enhance enterprise VR solutions. Not only is this technology now more immersive and accessible than ever, but it can also give retailers powerful insights into their customers' behaviours and preferences that they simply would not have access to via other mediums."

_{Francois de Bodinat, CMO, ZeroLight}

Impact at CES and subsequent showcases

The experience featured heavily in HTC's high-profile announcement, appearing in the keynote presentation where the flagship HMD was unveiled before being used as showpiece demonstration of its impressive capabilities on HTC's booth.

"VIVE Pro Eye is at the forefront of enterprise VR. Built-in eye tracking is going to forever change the way people do business. The BMW M Virtual Experience is proof. The project highlights the many unique benefits that VR brings to retailers and their customers. It played a vital role in the announcement of Pro Eye—we couldn't be prouder to be a part of it."

_{JB McRee, Director of Marketing - Enterprise Virtual Reality, HTC VIVE}

Hundreds of journalists and VIP guests were invited to try the demonstration across the five-day event, leading to significant reaction across the industry and press, including coverage in Engadget and Venturebeat. As a result of this success, the BMW M Virtual Experience has gone on to be shown at various international events in 2019 with both NVIDIA and VIVE, including Mobile World Congress, GTC (on two separate demonstrations), and Laval Virtual.