Visual Microphone Turns Images Into Sound

A team of researchers from MIT, Microsoft and Adobe developed an algorithm capable of reconstructing an audio signal by analyzing a video and detecting tiny vibrations that the surrounding sounds induce on the object.

[Image Courtesy of Abe Davis]

Each time a sound is made, the acoustic waves crash on objects around it creating tiny vibrations that are invisible to the naked eye. “When sound hits an object, it causes the object to vibrate,” said researcher Abe Davis, a graduate student in electrical engineering and computer science at MIT. “The motion of this vibration creates a very subtle visual signal that’s usually invisible to the naked eye. People didn’t realize that this information was there.”

The technology used in these experiments is similar to the ones in laser microphones which are used by spies to eavesdrop by measuring vibrations from reflective surfaces. But, instead of using expensive and sophisticated equipment, the team managed to turn all kinds of objects in microphones.

Normally, the technique requires high speed cameras (2000-6000 fps) when an ordinary camera captures only 60 frames per second – because the frequency of sampling a video must be greater than the frequency of the audio signal. However, the team found that a peculiarity in the design of CCDs, the sensors in digital cameras, makes it possible to extract information even from common cameras – enough for example, to distinguish a man’s voice from a woman’s voice.

In one experiment, understandable and clear sounds were obtained from the vibrations recorded from a bag of chips filmed 4.5 feet away and even through a soundproof glass. It was also possible to extract audio signals from videos of  aluminum foil, the surface of a glass of water and even music reflected on plant leaves.

Abe Davis says that he now wants to investigate whether the new technique can reveal information about the internal structure of objects. “Not only can we get information about the sounds emitted close to objects, but also within the objects themselves, because each one will respond to the audio in a different way,” he said.