Watch 3D movies at home without special glasses

But researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) aim to change that with "Home3D," a new system that allows users to watch 3D movies at home without having to wear special glasses.

Home3D converts traditional 3D movies from stereo into a format that's compatible with so-called "automultiscopic displays." According to postdoc Petr Kellnhofer, these displays are rapidly improving in resolution and show great potential for home theater systems.

"Automultiscopic displays aren't as popular as they could be because they can't actually play the stereo formats that traditional 3D movies use in theaters," says Kellnhofer, who was the lead author on a paper about Home3D that he will present at this month's SIGGRAPH computer graphics conference in Los Angeles. "By converting existing 3D movies to this format, our system helps open the door to bringing 3D TVs into people's homes."

Home3D can run in real-time on a graphics-processing unit (GPU), meaning it could run on a system such as an Xbox or a PlayStation. The team says that in the future Home3D could take the form of a chip that could be put into TVs or media players such as Google's Chromecast.

The team's algorithms for Home3D also let users customise the viewing experience, dialing up or down the desired level of 3D for any given movie. In a user study involving clips from movies including "The Avengers" and "Big Buck Bunny," participants rated Home3D videos as higher quality 60% of the time, compared to 3D videos converted with other approaches.

Kellnhofer wrote the paper with MIT professors Fredo Durand, William Freeman, and Wojciech Matusik, as well as postdoc Pitchaya Sitthi-Amorn, former CSAIL postdoc Piotr Didyk, and former master's student Szu-Po Wang '14 MNG '16. Didyk is now at Saarland University and the Max-Planck Institute in Germany.

Home3D converts 3D movies from "stereoscopic" to "multiview" video, which means that, rather than showing just a pair of images, the screen displays three or more images that simulate what the scene looks like from different locations. As a result, each eye perceives what it would see while really being at a given location inside the scene. This allows the brain to naturally compute the depth in the image.

Existing techniques for converting 3D movies have major limitations. So-called "phase-based rendering" is fast, high-resolution, and largely accurate, but it doesn't perform well when the left-eye and right-eye images are too different from each other.

Meanwhile, "depth image-based rendering" is much better at managing those differences, but it has to run at a low-resolution that can sometimes lose small details. (One assumption it makes is that each pixel has only one depth value, which means that it can't reproduce effects such as transparency and motion blur.)

The CSAIL team's key innovation is a new algorithm that combines elements of these two techniques. Kellnhofer says the algorithm can handle larger left/right differences than phase-based approaches, while also resolving issues such as depth of focus and reflections that can be challenging for depth-image-based approaches.

"The researchers have used several clever algorithmic tricks to reduce a lot of the artifacts that previous algorithms suffered from, and they made it work in real-time," says Gordon Wetzstein, an assistant professor of electrical engineering at Stanford University, who was not involved in the research. "This is the first paper that produces extremely high-quality multiview content from existing stereoscopic footage."

Didyk says that modern TVs are so high-resolution that it can be hard to notice much difference for 2D content. "But using them for glasses-free 3D is a compelling application because it makes great use of the additional pixels these TVs can provide," Didyk says.

One downside to converting traditional 3D video to multiview TVs is that the limited resolution can lead to images appearing with duplicates near or around them—a phenomenon referred to as "ghosting."

The team hopes to further hone the algorithm to minimise ghosting, but for now, they say that they are excited that their conversion system has demonstrated the potential for bringing existing 3D movie content beyond the multiplex.

"Glasses-free 3D TV is often considered a chicken-and-egg problem," Wetzstein says. "Without the content, who needs good 3D TV display technology? Without the technology, who would produce high-quality content? This research partly solves the lack-of-content problem, which is really exciting."