VR vs. AR: What will dominate the future?

Whether it’s because the technology is generally cumbersome, tethered or expensive, or because the visuals do not appear to be authentic enough, penetration is currently limited. This highlights just how far the technologies have to go before they can truly become ingrained in our day-to-day lives.

So what needs to change to take VR and AR to the next level, and ultimately, will one or both technologies succeed?

For the sake of my sanity and yours as the reader, I’ll keep to two classifications of headsets – VR headsets, where what the user sees is purely virtual and has no direct view of their surroundings, and AR headsets, which maintain the wearer’s direct view of the real world and overlay data onto that view.

As an aside, mixed and merged reality can be seen as subsets or supersets of the VR and AR headset categories (depending on your point of view). These devices basically overlay graphics onto real or virtual worlds. For example, the virtual image could include a camera feed of the real world. There is some overlap in the applications of AR and VR (and mixed/merged), but also quite significant differences.

Two of the main applications for VR today are gaming and immersive (360°) video playback. While gaming is also an application for AR, the focus of AR is more on professional and educational use cases.

We will of course see massive evolution of both technologies in the coming years, and therefore in the uses they are put to. As an example, whilst both could be used for viewing videos, in the case of VR it may be an immersive, private experience as opposed to AR that will seemingly project it onto your wall for an open experience.

What the numbers say about AR and VR
VR and AR technologies are rapidly evolving and growing, but the adoption rate is still very low. The latest figures are often skewed when quoted in the press by the inclusion of Google cardboard (or latterly Daydream) type devices, which use your smartphone for display and processing inside a dumb (or nearly) dumb optical headset. Whilst this use-case will definitely help drive the ecosystem, it is limited in driving hardware specifications and dedicated SoC evolution.

If we look at figures for dedicated VR headsets, and ignore the ‘opportunistic’ smartphone-based VR, we see that the market leader Sony shipped 429,000 units in Q1 2017, HTC shipped 190,000, and Facebook (Oculus) shipped 99,000 units, for a total of 719,000 units. For context, this is around 0.1% of smartphone shipments in the same period, or around one percent of the shipment of smart TVs in the US alone in the same period (the VR headset figures are worldwide). These volumes for shipments of VR headsets are, by any measure, tiny.

Above: There are some good demos out there now which have evolved beyond Pokémon Go

In case you are wondering why I haven’t spoken about AR headsets yet, according to ABI Research, the total shipment of AR headsets in 2016 was only 75,000 units or a barely registering, 0.005% of smartphone shipments.

We need to be clear that AR and VR are nascent technologies, and as ever we need to let the technology advances catch up with our imaginations. One of the best analogies I have seen on the web, as well as being an interesting read on VR in general, is that by DPReview author Dale Baskin. He compares the current generation of VR headsets to an Osborne 1 portable computer and clearly we have a way to go in terms of hardware. Whilst VR headsets may have some way to go in terms of form factor and functionality, at least the AR headsets have some fairly sleek form factors available, as the recently released Google Glass Enterprise Edition. However, the performance and battery life of these devices are still limited.

Predicting the future of AR and VR
As an IP provider, the PowerVR Business Unit of Imagination Technologies is typically working three to five years ahead of products being available for end users. This means that we have to predict the requirements very early for markets that are still yet to mature. For some time now we have been looking at what will drive the requirements for VR and AR over the next few years.

There is a relatively easy split in terms of VR and AR Head Mounted Displays (HMDs) – those that do the majority of processing on-board (standalone HMDs), and those that do the processing off-board (tethered HMDs), connected to the processing unit via a cumbersome cable or a wireless link, such as those by display link. Due to the lack of freedom of movement a wired tethered solution gives, looking forward I expect the mainstream solutions to be tethered wirelessly, (which needs to overcome the latency and transmission bandwidth issues), or standalone.

Taking VR first, how do we as an industry advance? There are many reasons for the relatively sluggish VR uptake, but we must put user experience at the top of that list. If the user experience was great, then the obstacles of cost and market fragmentation, as well as form factor, would all be easily overcome. The main issues with the user experience of headsets today are in relation to low resolution, inadequate frame rates and colour depth, motion to photon latency (the time it takes the display to update from any movement), and form factors. In terms of tethered headsets, as I have mentioned, I expect we will transition to mainly wireless solutions, with power consumption a key issue to overcome.

We need to see a step-change in resolution for both standalone and tethered headsets. The current market leading headsets have a resolution of around 1Kx1K per eye, with a field of view of around 110°. Samsung is upping the resolution in the next generation GearVR to over 2Kx2K per eye, which gives a noticeable improvement in quality. The general industry consensus seems to be that a resolution of 4K per eye, at 120Hz should give a worthwhile boost in user satisfaction, although we are seeing some companies targeting much high resolutions.

This includes start-up Varjo who have announced they are working on a 70MP headset. However, as this is an early stage start-up with no shipping product, we must patiently wait for real products. Whilst 70MP may sound like a ridiculously high resolution, in fact it only equates to just over 8K per eye, so in reality is not so hard to believe. A white paper by microprocessor and semiconductor expert David Kanter suggests that for the ‘retina’ VR experience, a resolution of over 100Mpixels/eye is required. The reasoning for this number is explained in the paper and covers the interesting discussion on pixels per degree (PPD) rather than using the more traditional pixels per inch (PPI). The demand for these ultra-high resolutions, may well push us towards foveated rendering solutions, for which the PowerVR ray-tracing GPU is ideally positioned (more on that below).

What does this increase in resolution mean for Imagination as a GPU vendor? Well, it is unlikely that we (or any other GPU vendor) will be creating 70MP graphics at 120fps any time soon. The expected evolution to 4K/eye at 120Hz already represents an increase in pixel throughput of well over 2,500% compared to today’s solutions (the 70MP is 11,500% more).

These high fill rates are achievable today, as is the processing needed to create compelling content. The main issue is power. To achieve the required performance level, a dedicated GPU is required, specifically designed from the outset to meet these needs. Desktop systems are beginning to be capable of delivering this level of performance, but clearly need massive power supplies and cooling devices. For a standalone system today, the only IP option for achieving close to this high level of performance in a constrained power footprint is a PowerVR Furian based GPU. The PowerVR Furian 8XT family can deliver sustained performance at high resolutions and frame rates. If an SoC vendor were to build a dedicated AR/VR headset SoC with an 8XT GPU, they could achieve a significant increase in performance vs. the typical re-use of the mobile phone (or even tablet) application processors that we see today.

Above: AR will see far more B2B focussed applications, such as warehouse operatives identifying items, aircraft engineers having complex circuitry overlaid onto the engines to explain service procedures and surgeons who can identify potentially cancerous cells

Ideally, we would go further than this with a dedicated VR SoC, in order to optimise the power efficiency, and use a PowerVR ray-tracing GPU, to perform some of the required tasks such as lens correction, async time warp and eye-tracked foveated rendering. These techniques running on a PowerVR ray-tracing GPU will deliver significant power and performance benefits over traditional raster-based rendering techniques. And of course the ray-tracing unit can be used to generate significantly higher quality graphics. By virtue of its ability to produce photo-realistic shadows, real time ambient occlusion modelling and real refraction effects to name a few, ray tracing can bring about a visual fidelity that is just not possible with traditional raster-based graphics. Ray tracing efficiencies also enable us to keep a much higher raster graphics fidelity without which we would suffer the issues of solutions today that need to reduce high end features to achieve the required latency.

Moving on to AR, we can make arguments either way in terms of the quality of the imagery. It can be argued that since we are overlaying graphics onto the real world, and not viewing an entirely computer generated scene, the images can be more basic. Of course, we can also make the argument that seeing as how we are overlaying graphics onto the real world, we need them to look photo-realistic, or they will be easily spotted as fake. There are some good demos out there now which have evolved beyond Pokémon Go, such as the recent real world Mario Brothers game, created by Abhishek Singh, running on the HoloLens.

If we want to extend these AR experiences to items that look far more like actual objects, we are going to need some form of SLAM (Scene Localisation and Mapping) technique to tell us where the items are in the real world scene. Once we have mapped the scene and identified the objects in it, typically using some form of machine learning (AI/CNN), we need to understand the lighting in the scene to be able to accurately measure the shadows and accurately portray the items. It’s amazing how adept the human brain is at picking up small, seemingly insignificant details that tell us something is fake. With Imagination’s solutions, we can use the GPU to provide additional computational power for the aforementioned algorithms, and dedicated hardware for the most power efficient and performant solutions.

Is there a winner? 
The industry has some way to go for both AR and VR to produce really compelling solutions for end users. We have a traditional chicken and egg situation, with the current low volumes limiting the investment that companies are willing to make. It will undoubtedly take something big for the companies to make that initial investment in dedicated SoCs for standalone HMDs. Without the step change in performance, AR and VR may fail to ever achieve their true potential.

I am convinced that when VR headsets are able to deliver a true ‘Avatar’ type experience, where users can really be fooled into believing they are inside an alternative reality, then the adoption of HMDs will soar. Use cases will be for applications as diverse as travel agents showing holiday destinations to potential clients, as a relaxation tool, for estate agents giving virtual tours of new builds, education and of course, the ubiquitous gaming.

AR will see far more B2B focused applications, such as warehouse operatives identifying items, aircraft engineers having complex circuitry overlaid onto the engines to explain service procedures and surgeons who can identify potentially cancerous cells, to name but a few. However, in order for this explosion of use cases to happen, the intelligence, form factor and battery life will need to be improved.

So, ultimately it won’t be ‘AR or VR’, but ‘AR and VR’, with each finding the applications it suits best, and as the industry and technology develop, we will see adoption of both accelerating.