Camera-based assistance system handles inattentive driving

This system detects whether the driver is showing signs of drowsiness (nodding off) or is distracted. And the car instantly responds – with a vibrating seat or a warning tone, for instance. The less inattentive the driver gets, the more attentive the car gets. In order to respond quickly and accurately, the assistance and the emergency braking systems can automatically activate in advance of a potential emergency.

“For semi-autonomous or autonomous driving, the assistance system of the vehicle needs to know the state of the driver at all times,” says Frank Blaesing, Head of Innovation Management and Advanced Engineering at Kostal. “It needs to know if and how well the driver is informed about what’s happening on the road at that moment. This information is especially needed when the control over the vehicle needs to be turned back to the driver in a complex traffic situation.”

“We want to make driving safer for all road users,” says Jochen Hanebeck, who heads the Automotive division at Infineon Technologies. “Our chips help to reduce the number of serious road accidents. For us, the collaboration with Kostal is a significant contribution to assisted driving, with all its safety and comfort advantages.”

By 2018, cars with the 3D camera system from Kostal will be running off the assembly line. The optical system is only 49 by 29mm in size and is embedded into the dashboard: Through the steering wheel, it “looks” front-facing at the driver’s body and head. It records the exact head position and recognises the blink of an eye even through glasses or sunglasses.

The 3D camera uses infrared light and works even under changing light conditions or in the dark. It contains the Real3 3D image sensor chip from Infineon, which reliably records more than 100,000 pixels 50 times a second. For each pixel it simultaneously recognises the distance and the brightness value. From the depth image of the distance data and the amplitude image of the brightness values, the 3D camera system then identifies 49 predetermined points in the driver's face such as eyes, eyebrows, mouth or tip of the nose. Algorithms use this data to calculate the driver’s attentiveness. If, for example, the system detects signs of fatigue in the driver, it initiates the countermeasures.

In addition, knowledge of the exact position of the driver’s head helps to project navigation information in the windshield through HUDs, so that it’s right in the driver’s line of sight, seamlessly fitting into the streetscape in front of him. The driver will thus always see the navigation arrow in front of him at the same place in the road – regardless of how he holds his head.

The heart of the camera system is the 3D image sensor chip. It is based on the Time-of-Flight (ToF) principle, which measures the time it takes for the infrared light to go from the camera to the object and back. This time relay is directly related to the distance of the object. Compared to other 3D measurement methods, with ToF, the depth data is measured directly and doesn’t need to be determined via complex algorithms. Furthermore, at the end of their production process, 3D cameras with Real3 can be calibrated easily and permanently and only need one lens (monocular camera architecture) to be completed. Therefore, they are relatively small and impervious to vibrations in the vehicle.

Because of their high data quality, 3D ToF cameras aren’t in the vehicle just for detecting head position and blinking. They can be used for controlling infotainment systems or the air conditioning with hand movement or a body gesture. For example, the vehicle could set the airbags individually for each passenger. In addition, assistance and safety functions outside the vehicle are possible, such as a door opening assistance that prevents you from bumping the car door against another car, wall or ceiling when parking at a commercial parking garage or at home.

With Infineon as one of its technology partners for the future of driver assistance, Kostal is presenting its 3D camera system for driver monitoring and other innovations under the motto “Mechatronics for the People!” in Hall 8 at Booth A01 at the 66th International Motor Show (IAA, 15^th to 27^th September 2015, Frankfurt am Main, Germany).