Augmented reality: a next-gen approach to automotive displays

This article originally appeared in the December'24 magazine issue of Electronic Specifier Design – see ES's Magazine Archives for more featured publications.

Today’s software-enabled vehicles collate and share data from multiple sources to support those within the cabin, with one of the most prevalent use cases being GPS navigation. But even with today’s technology, this can cause safety issues.

To follow the navigation, a driver’s eyes often need to flit between the road and either a central navigation display or cluster display, or even a navigation app on their smartphone. This means that not only are the driver’s eyes off the road for multiple seconds at a time but transposing that mapping information onto the road ahead increases the cognitive demand placed on the driver, which can contribute to distracted driving. 

In 2022, an estimated 458 fatal accidents were caused by distracted driving in the UK (DfT). That number exceeds 3,000 fatalities annually in the USA (NHTSA) and is the number one reason for accidents. While the display cluster may be today’s standard for sharing this information in modern vehicles, it is not meeting the complex requirements to best support the driver and safety.

Augmented reality bolsters recognition

Seeking a solution to better integrate navigation guidance and a driver’s real-world view, HARMAN’s engineering team focused on integrating head-up display (HUD) technology with AR. While HUDs have been around for some time, progress has been slow as they can be overly complicated for users, creating cluttered interfaces that have the potential to overwhelm. Meanwhile, OEMs are seeking solutions that don’t compromise on package volume, cost and power efficiency – especially considering the shift to electric vehicles.

HARMAN’s ultimate goal was to create a transparent HUD solution integrated into the windshield that’s equipped with AR to deliver an intuitive experience. The result is the HARMAN Ready Vision AR HUD, a highly precise and intuitive solution that empowers the driver with the right information, at the right time and in the right way to help reduce distraction, manage cognitive load and improve safety.

This advanced AR HUD product is unique in the marketplace and already available in the latest all-electric Porsche Macan and Audi Q6 e-tron models.

Synchronising data

HARMAN's Ready Vision AR software combines all available sensor inputs in the vehicle and extrapolates them to build a 3D environmental model that includes directional alerts, ensuring drivers can rapidly recognise, comprehend and react to the situation around them.

Several key technical challenges had to be overcome during the process. Sensor data, such as vehicle speed and Inertial Measurement Unit (IMU) readings for example, and camera data have different latencies and frequencies that needed to be synchronised. High-frequency shake compensation was essential to ensure that vehicle movement does not cause noticeable delays or distractions for the driver, which could result in dizziness or fatigue. Additionally, the human machine interface (HMI) and visualised features need to be clear and concise while remaining practical; they should not cover critical objects such as traffic signs or other road users or obstruct the driver’s view.

Environmental fusion reduces latency

Combining data extrapolation and prediction in our environmental fusion model with high-frequency pitch shakes compensation, the AR software reduces object recognition time to the best minimal latency currently available in the market.

Typically, it takes approximately 100 milliseconds until an object captured by camera sensors is displayed on the HUD, with an additional 50 milliseconds needed for the driver to recognise the object. Yet with environmental fusion, we can use Global Navigation Satellite System (GNSS, latency 70ms) and Inertial Measurement Unit (IMU, latency 5ms) data and combine it with the signals from the vehicle’s sensors (latency 20ms). Consequently, we can iterate objects and their future position and display this projection on the vehicle’s HUD, achieving a total latency of less than 50 milliseconds. 

To accomplish this, HARMAN’s AR Creator handles over 1,500 messages, processing approximately 200KB of input data per second from various vehicle sensors. It continuously analyses this information, fusing it into an environmental model. This model is then used to implement the positions and behaviours of augmented objects and represent 2D features, all at a smooth 60 frames per second.

Minimising development time and cost

With time to market being critical for OEMs, the software is built to automotive grade and aligned with certification standards, with the core elements reused across vehicle platforms with minimal testing requirements between model variants for rapid development.

Our design team has also created a tool that includes a user interface editor, so that when development moves to in-vehicle testing features and appearances can be altered in near-real time to find the best possible solution for that vehicle. This includes simple adjustments such as colour, but also layout changes, sizes and behavioural parameters. This approach helps the teams iterate quickly – sometimes in as little as 15 minutes – to minimise testing time. Additional features can also be added with few to no code changes, so the rollout of OEM preferred individual features is rapid and seamless. 

Toolkits for software development and AR are provided to customers, which enable in-house developers to create branded interactions or implement software updates as needed. This either streamlines or negates the need for in-vehicle testing ahead of update implementation, saving OEMs time and money.

Rollout of safety advances

At present, HARMAN’s AR HUD is featured in select Porsche and Audi models and is more accessible to vehicles in the premium and luxury segment due to the price of implementation. However, in a move to enhance driver safety and improve the user experience in a broader range of vehicle models, HARMAN has developed Ready Vision QVUE, which takes advantage of the unused space at the base of the windshield, reflecting driving information closer to the driver’s line of sight without obstructing the road in a cost-effective way.

As the same software it used across the AR display and QVUE solution, OEMs opting for QVUE in the short term due to budget constraints will not have to duplicate all of the features and functionalities once they add or switch to AR displays – helping save money at the next step in development. Or, if space is available in the dashboard, the integration of both would create the best solution for enhanced driver safety.

In such a setting as in-vehicle cabin displays, augmented reality offers huge advantages in terms of driver focus and safety, with the potential to have a major impact on the reduction of GPS-related and distracted driving incidents.