3d images

This ‘lensless’ camera captures 3D images in a single shot – and can even see ‘through’ obstacles

A pair of researchers from the University of California, Davis have created a “lensless” camera capable of capturing three-dimensional images in a single shot, with full refocusing capabilities after capture – using a neural network to improve real-time reconstruction performance. time without complex calibration.

“We consider our camera to be lensless because it replaces the bulk lenses used in conventional cameras with an array of thin and lightweight flexible polymer microlenses,” says team leader Weijian Yang. “Because each microlens can observe objects from different viewing angles, it can perform complex imaging tasks such as acquiring 3D information from objects partially obscured by objects closer to the camera.”

Based on technology the team originally developed for a microscope, the microlens array captures multiple views of the same scene in a single pass. Although each sub-image only contains two-dimensional data, the angle difference between each microlens means that 3D data can be computed – with a newly developed neural network that does this in real time.

“Many existing neural networks can perform designated tasks, but the underlying mechanism is difficult to explain and understand,” Yang says. “Our neural network is based on a physical model of image reconstruction. This makes the learning process much easier and results in high-quality reconstructions.”

Images captured by the camera system can be recentered after capture, while the neural network also produces a depth map, which can be used for navigation or 3D modeling tasks. The camera can also see “through” opaque objects, making them transparent provided at least one microlens can see around them – a world first for lensless cameras, Yang says.

“This 3D camera could be used to give robots 3D vision, which could help them navigate 3D space or enable complex tasks such as manipulating thin objects,” Yang says. “It could also be used to acquire rich 3D information that could provide content for 3D displays used in games, entertainment, or many other applications. With the recent development of advanced micro-optics manufacturing techniques and at low cost along with advances in machine learning and computational resources, computational imaging will enable many new imaging systems with advanced functionality.”

The team is now working to improve the image quality by reducing the error rate and miniaturizing the hardware in order to be able to integrate the technology into future smartphones.

The work of the pair is published in the journal Express Optics in free access.