In accordance with the proliferation of IoT (Internet of Things), there is a growing need for a camera that is so unprecedented that it can be installed anywhere. To address this need, Hitachi has developed a "lensless camera" that can take photos and video images without using a lens.
Eliminating the lens has helped realize a very thin and compact model. Moreover, the Company's proprietary photographing technologies have made it possible to enhance the processing speed and adjust the focus after shooting the images. The lensless camera, which features thinness, compactness and high functionality, will bolster the evolution of IoT going forward.
TAJIMA Kazuyuki
Researcher
SAO Mayu
(Publication: December 1, 2017)
TAJIMACameras usually have a lens without exception, so it's no wonder that this is puzzling.
Conventional cameras have a lens and an image sensor as a set. Light is collected through the lens to form an image of the photograph subject. The image is then projected on to the image sensor. That's how conventional cameras work.
In contrast, the lensless camera uses a permeable film instead of a lens. The film is imprinted with specific patterns such as fringe patterns. Unlike the lens, the film does not form an image. It just projects the shadow that is made when light passes through the film. The projected images are nothing but fringe patterns, and human eyes cannot tell what images have been taken. Therefore, the lensless camera conducts signal processing to convert the shadow into an image that is visible to humans. The captured image is reconstructed in this manner. What characterizes the lensless camera is that it combines an optical technology of projecting shadows and a signal processing technology of restoring the shadows into captured images.
Figure 1: Structure of the lensless camera
TAJIMAAs IoT is spreading quickly, Hitachi is focusing on developing technologies related to IoT. One of the issues in developing the IoT technologies is how to collect the information of specific "things" to be connected to the internet in terms of their physical features. In this regard, there has been a need to use a camera in places where cameras could not be used, such as setting cameras in gaps of pipes to examine details. To set a camera in such a gap, a camera that is much thinner and smaller than before is needed. In addition, the production costs of such a camera must be reduced so that they can be placed in a variety of locations. To do so, however, a very thick and expensive lens is an obstacle.
With the times, cameras have achieved higher functionality with higher picture quality. However, the concept of using a lens has not changed for a long time. In order to develop a new camera that is different from conventional ones, innovation with regard to the lens concept was required. That's what we thought.
SAOA lensless camera's photographing principle of using a permeable film instead of a lens has been studied by other companies as well. To date, such a lensless camera has needed a complicated signal processing to reconstruct the captured image, which takes a lot of time for calculation.
A camera for collecting information to be used in IoT must send captured images and video images as quickly as possible. Therefore, aside from realizing a thinner and more compact model by eliminating a lens, the signal processing must be simplified and performed more quickly.
TAJIMAThe new camera has largely two distinctive features. One is that its signal processing has become simpler and faster as a result of elaborating on the patterns printed on the permeable film.
In order to simplify and quicken the processing, we took note of what is called "Moiré fringes." These are striped patterns that appear when orderly aligned and repeated patterns are overlapped, and are often generated in processing pictures and video images. We noticed from analyzing the Moiré fringes that we can learn the direction and angle of the light.
So we decided to print patterns of concentric circles on the permeable film used for the lensless camera. Patterns of concentric circles are overlapped to generate Moiré fringes, which are then analyzed to reconstruct the image. That's the idea of how it should work.
SAOTo analyze the Moiré fringes, we utilize the Fourier transform, which is often used for processing pictures and video images. By using an analysis method that has already been widespread, we were able to reduce the volume of calculations necessary for the processing. Indeed, the calculation volume was remarkably reduced by a digit or two compared with the lensless cameras developed to date. Because the processing is simpler, the newly developed camera can capture not only still pictures but also video images.
Figure 2: Photographing principle of Hitachi's lensless camera
TAJIMAThe other feature is that the camera has achieved a function to adjust focus to any preferred position even after shooting the images by generating the Moiré fringes through digital processing.
Images that have Moiré fringes can be captured by placing two permeable films in front of the image sensor. However, that only tells the direction and angle of the light for a particular focus. So we decided to superimpose the pattern of concentric circles on the second film through digital processing. By changing the way the patterns are superimposed through digital processing, the focus can be freely changed. Moreover, the calculation volume is so small that real-time processing is possible. This means that the focus can be adjusted while shooting video images.
SAOWhat matters is that well-known phenomena and popular methods are used to achieve the two features. For reconstructing images through computation, we employed computational photography, a technology which is now the center of attention. There is a strong impression that calculations in computational photography are extremely complex. But Hitachi's technology can sweep away such an impression.
TAJIMAFrankly, we were not necessarily sure. We first tried to take a photo of a line, and created a verification model. But we were not able to capture a picture of the line at all.
Now we can tell you why. For example, one of the causes was the distance between the image sensor and the permeable films. The distance is an important factor for a lensless camera. The optimum distance is a few millimeters. Otherwise a picture can never be captured. However, when we started verification, we did not know that and the distance was as far as two centimeters. We made a verification model and failed to take a picture, and repeated this cycle over and over. We were quite uneasy, doubting if we could really be successful.
When we finally succeeded in taking a photo of a single line, we were deeply impressed and were convinced, for the first time, that we could be successful.
SAOEven after the picture of the line was taken, it seemed that it was hard to attain a fine picture of it.
When Tajima and his colleagues were working on the verification of the photographing principle, I was not yet involved in the project. When I saw the pictures at that time, I secretly doubted if they would be successful. I knew it was difficult to reconstruct images through calculation, but the picture quality was too bad (laugh).
TAJIMAAs we had succeeded in taking a photo of a line, we then tried to take photos of drawings. But the captured images were very bad. They were awful and looked like ghosts.
In our efforts to be able to take fine pictures, we also shared the verification results with members of other projects and held many discussions with everyone on what needed to be fixed. Thanks to the efforts of all the people involved, we were able to internally present the research results in such a short time of about two months after starting the investigation, and announced a news release within a year.
SAOWe also focused on selling the technology so that the new photographing technology can be used widely. While we worked on technical investigations, we introduced the lensless camera technology to various people within the Hitachi Group and asked them if it could be used somewhere. When we talked with them, we gave a demonstration using the actual model. It appeared they became more interested when they actually saw the camera and the photographing results.
That is why we worked hard to create an actual model to show it to customers for demonstrations. By appearance, it matters that a lensless camera can be thinner and more compact. To shoot photos in the laboratory, we used a verification model that incorporated a very large image sensor. But we could not show it to customers as is, because they would doubt our appealing the compactness of the camera. So we endeavored to create the actual model to be unveiled at the time of the news release so that the thinness and compactness of the camera could be appealed.
Figure 3: The lensless camera unveiled with the news release
TAJIMAWe were surprised by the greater than expected reactions that were received. Before we announced the news release, we wondered if people would take time to come and see it. Once the news release was delivered, however, we received many inquiries not only from the press but also from such outside people as those engaged in camera-related fields and manufacturers making film used for lensless cameras. When we made a presentation at an academic conference after announcing the news release, many people came from far and wide to listen to us. The reaction was larger than anything we had ever experienced.
SAOBecause a camera is a familiar product, many people were interested in the technology. In particular, I was very happy to find that ordinary people were interested in it.
TAJIMAMy first priority is to deliver the lensless camera technology to ordinary people as a product. Since joining the company, I've had a dream of seeing products that I have developed myself to be sold in appliance stores in town. So I hope that this dream comes true. I want to make the lensless camera as popular as conventional cameras. I would be delighted if it is widely used in applications where its thinness and compactness advantages are valued.
Computational photography is an extremely interesting technical field, and is a theme that has just entered a rising tide. I also want to develop new technologies that utilize the experiences I have gained with the lensless camera.
SAOThe lensless camera is technically unique and very interesting. Researching the technology is very satisfying. Like Tajima, I want to continue discussions on commercializing the technology and creating related services, while conducting research to enhance and fine-tune the lensless camera technology itself.
This research was started in the year I joined the company, and I was engaged in different research work at that time. I gained a desire to be involved in the research of the lensless camera when I saw Tajima and his colleagues work on the research. Luckily, I was able to participate in the project from my second year at the company. What is more, we were able to create research results that received great response. This really moved me.
In the research, we received advice from so many people, and everyone joined hands to produce the results. As many people cooperated in the project, I will continue my endeavors in a manner that will allow me to express my appreciation for their help.
