Fujitsu Develops Eye Tracking Technology

Details on this technology will be exhibited at CEATEC Japan 2012, opening October 2 (Tuesday) at Makuhari Messe in Chiba, Japan.



Background

Eye tracking technology that can grasp where a user is looking and manipulate a PC screen—such as automatic zooming or scrolling—has attracted high expectations as a new interface that requires little effort on the part of the user. This kind of interface, however, had required a large and expensive specialized camera as well as an LED, limiting actual usage applications and preventing its widespread adoption to ordinary PCs.

##IMAGE_2_C##

Figure 1. Example of the Eye Tracking Interface



Technological Issues

The corneal reflection method, which uses an LED to emit near-infrared light and a camera to track the reflection on the user's eye, presents a non-invasive, highly accurate way to calculate the direction of an individual's line of sight. In this method, which negates the need to wear goggles, the image of the eye captured by the camera is analyzed to calculate where a person is looking. It tracks the pupil's movements that change depending on the direction of the user's gaze, and also detects the invisible near-infrared light from the LED that is reflected off the cornea. The position of the reflection is not affected by the line of sight, so the direction of the gaze can be calculated from the positional relationship between the pupil and the corneal reflection. This method requires accurate detection of the corneal reflection and the middle of the pupil from the camera image, which is why up until now large, expensive, high-performance cameras and LEDs were needed.



Another issue is that cameras and LEDs need to be embedded into a PC or other consumer product in a way that is both inexpensive and that does not detract from the PC's design. When using a low-cost, compact camera and LED, however, the sensitivity of the camera and the strength of the LED are low, resulting in a blurry image of the pupil and a low degree of corneal reflection. As a result, it is difficult to accurately detect the pupil and the corneal reflection and calculate their position from the image taken.

##IMAGE_3_C##

Figure 2. How the Corneal Reflection Method Works



The Newly Developed Technology

Fujitsu Laboratories has developed eye tracking technology that is small enough to be embedded in a PC using a compact, inexpensive camera and an LED. Using a near-infrared LED and the type of general-purpose, compact camera that is already widely embedded in PCs, an eye tracking module with an overall width of 7mm was created (see figure 3). In addition, the processing required to detect an individual's line of sight from the image captured by the camera is accomplished using software, minimizing hardware costs. The features of this technology are described below.

##IMAGE_4_C##

Figure 3. Structure of the Eye Tracking Module



1. Extracting pupil and corneal reflection candidates

The image captured with the compact camera is run through image processing software. As the pupil is darker than the rest of the eye and, conversely, the corneal reflection is lighter, the software uses this characteristic to extract candidates having these qualities from the eye area. The size of the pupil varies in accordance with ambient light, and that is taken into account to extract candidates of varying sizes in order to prevent viable candidates from mistakenly going undetected.



2. Narrowing down to the correct candidate

From the candidates extracted, rules for consistency are applied regarding the relationship between the camera, LED, and the position of the face, as well as with such factors as the size of the pupil just before the image capture, in order to narrow down to the candidate with the highest degree of consistency. For example, one rule is that the pupil size does not ordinarily change significantly from one image to the next during image processing. This ensures the image of the pupil size will not differ greatly from right before the image was taken to right after, so the difference will also not be very significant. As a result of this type of image processing, the pupil and corneal reflection are accurately detected, and from their positional relationship, the direction of sight is calculated. This process enables accurate detection of the pupil and corneal reflection even when the condition of the image is poor, thereby enabling the line of sight to be calculated.

##IMAGE_5_C##

Figure 4. Overview of Detection Method



Results

Using this technology, a non-contact sensor that detects a person's eye movements can be embedded in a PC without detracting from the PC's design. The result is a new interface that can automatically scroll or zoom the screen based on where a user is looking. Moreover, because it can detect where on a screen a user is looking, it opens up the potential for applications based on an understanding of what catches a user's attention and interest.



Future Plans

Fujitsu Laboratories is proceeding with the verification of this technology with the aim of commercializing it within fiscal 2012. It will also conduct further R&D on new interfaces and other application technologies that take advantage of eye tracking capabilities.