UW researchers develop headphones prototype

The team’s artificial intelligence algorithms, combined with a headphone prototype, allow the wearer to hear people speaking within a bubble with a programmable radius of 3 to 6 feet. Voices and sounds outside the bubble are quieted an average of 49 decibels (which can be understand as roughly the difference between a vacuum and rustling leaves), even if the distant sounds are louder than those inside the bubble.

Findings from the research were published on 14 November in Nature Electronics, with the proof-of-concept device available for others to build on. 

“Humans aren’t great at perceiving distances through sound, particularly when there are multiple sound sources around them,” said senior author Shyam Gollakota, a UW professor in the Paul G. Allen School of Computer Science & Engineering. “Our abilities to focus on the people in our vicinity can be limited in places like loud restaurants, so creating sound bubbles on a hearable has not been possible so far. Our AI system can actually learn the distance for each sound source in a room, and process this in real time, within 8 milliseconds, on the hearing device itself.”

The prototype was created with noise-cancelling headphones, with small six microphones fixed across the headband. 

The neural network, which runs on a small onboard embedded computer attached to the headphones, tracks when different sounds reach each microphone. The system suppresses the sounds coming from outside the bubble, while playing back and slightly amplifying the sounds inside the bubble.

“We’d worked on a previous smart-speaker system where we spread the microphones across a table because we thought we needed significant distances between microphones to extract distance information about sounds,” added Gollakota. “But then we started questioning our assumption. Do we need a big separation to create this ‘sound bubble’? What we showed here is that we don’t. We were able to do it with just the microphones on the headphones, and in real-time, which was quite surprising.”

Training the system so it created sound bubbles in different environments with different stimuli, researchers required a distance-based sound dataset collected in the real world, which was not available. To gather this dataset, the headphones were placed on a mannequin head. A robotic platform rotated the head while a moving speaker played noises from different distances. As a result, the team collected data with the mannequin system and with human users in 22 different indoor spaces including offices and residential. 

The researchers determined that the system works for a couple of reasons: the wearer’s head reflects sounds, allowing the neural net to distinguish sounds from various distances, and sounds have multiple frequencies, each of which goes through different phases as it travels from its source. 

The team’s AI algorithm, the researchers believe, is comparing the phases of each of these frequencies to determine the distance of any sound source (a person talking, for instance).

Headphones such as Apple’s Airpods Pro 2 can amplify the voice of the person in front of the wearer while reducing background noise, which works by tracking head position and amplifying the sound coming from a specific direction rather than gauging distance. 

This means the headphones can’t amplify multiple speakers at once; lose functionality if the wearer turns their head away from the target speaker; and aren’t as effective at reducing loud sounds from the speaker’s direction.

The system has been trained to work only indoors, because getting clean training audio is more difficult when outdoors. In the near future, the team is working to make the technology function on hearing aids and noise-canceling earbuds, which requires a new strategy for positioning the microphones.