Software enables mobile devices to sense pressure

The developers envision these and many other uses for their technology, which could offer the masses a coveted feature of the latest generation of smartphones.

The iPhone 6s has a force-sensing screen but the less expensive iPhone SE does not. No commercially available device has a pressure-sensitive body. "You don't need a special screen or built-in sensors to do this. Now this functionality can be realised on any phone," said Kang Shin, the Kevin and Nancy O'Connor Professor of Computer Science in the U-M Department of Electrical Engineering and Computer Science.

"We've augmented the user interface without requiring any special built-in sensors. ForcePhone increases the vocabulary between the phone and the user."

Shin created the system with Yu-Chih Tung, a doctoral student in the same department. "I think we're offering a natural interface, like how you turn a knob," Tung said. "It's the next step forward from a basic touch interface and it can complement other gestured communication channels and voice."

ForcePhone works by borrowing two of a phone's fundamental attributes—its microphone and speaker. The software sets the speaker to emit an inaudible tone at a frequency higher than 18 kHz, which is outside the range of human hearing. But the phone's mic can still pick up the vibration caused by the sound.

When a user presses on the screen or squeezes the phone's body, that force changes the tone. The phone's mic can detect that, and the software translates any tone tweaks into commands. 

"Having expensive and bulky sensors installed into smartphones can solve every problem we have solved, but the added cost and laborious installation prevent phone manufacturers from doing it," Tung said. "Our sound-based solution can fill this gap, providing the functionality without making any hardware modification. Everything is just software."

The idea of harnessing the phone's microphone and speaker for other purposes is an approach Tung initially picked up from the 2008 Batman movie, "The Dark Knight." In the film, Batman turns all the smartphones in Gotham City into a sonar system as high-frequency audio signals bounce off the city's infrastructure. He uses them to track the Joker.

"I thought it was an interesting idea to turn smartphones into a sonar-based system and felt this could lead to new applications to address challenges faced by smartphone users," Tung said.
ForcePhone is the third piece of software based on this idea. Previously, Tung and Shin created BumpAlert, an Android application designed to warn distracted pedestrians of objects in their path, and EchoTag, which let phones tag and remember specific indoor locations and associate those places with certain modes or tasks.

BumpAlert uses the phone's speaker and microphone to create an acoustic detector; its accelerometer to estimate motion and speed; and its rear camera to identify dangerous objects in the path. An algorithm combines the data and warns walkers by vibrating.

EchoTag works by sending out inaudible tones and picking up how they're reflected. It can do things like turn on an alarm clock or enable silent mode when you set a phone on a nightstand at the end of the day, for example.

While BumpAlert and EchoTag used sonar to create new applications for smartphones, ForcePhone builds more fundamentally on core interactions a user can have with a phone.

Shin is a leader in developing applications for mobile devices. Software he developed called E-MiLi, which stands for Energy-Minimising Idle Listening, could extend smartphone battery life by as much as 54 percent. Another system called GapSense helps reduce dropped calls and wireless device interference.