The industrial sector always seeks to secure high-level quality control and to enhance production efficiency.
In response to these requirements, Hitachi, Ltd. together with Daicel Corporation has developed an image analysis system that comprehensively analyzes the three factors (man, machines and materials) at plants. With a focus on "people," the new system uses cameras to sense workers' movements and, through image analysis, detects activities that deviate from the norm. This function makes it possible to identify human-caused problems at an early stage, which has been difficult to perform.
Utilizing the system, Hitachi will innovate the frontlines of manufacturing through "co-creation" with its customers.
NAGAYOSHI Hiroto
Senior Researcher
TERADA Takuma
Researcher
(Publication: November 2, 2017)
NAGAYOSHIYes. Jointly with Daicel Corporation, which manufactures airbag inflators for cars, etc., we have developed an image analysis system that detects deviations in worker activities at the frontlines and signs of operational failures in production line facilities. The system installs five types of cameras, comprising range cameras, fixed cameras, high-speed cameras, PTZ (Pan, Tilt and Zoom) cameras and omni-directional cameras, at plants. The cameras take images of the situation of workers, facilities and material processing. The images are analyzed to detect differences from normal situations, and the results are informed to supervisors. As such differences can be a sign of mistakes or failures, this function supports early detection of failures as well as improvement of quality and enhancement of productivity. The system has been created based on the production know-how of Daicel and by applying the image analysis technology fostered by Hitachi.
Figure 1: Configuration of image analysis system
TERADAOf this mechanism, we were mainly engaged in the "detection of deviations in worker activities" that focuses on human movements. Of the installed cameras, the range cameras can obtain three-dimensional shapes of the objects to be photographed. For persons in particular, we can estimate the position information of joints such as hands, elbows and shoulders from the three-dimensional shapes. We utilized this information to develop the technology to detect the movements of workers that are different from standard movements (deviations in worker activities).
NAGAYOSHIEven before the development, Daicel had a belief that "human movements" on the frontlines of manufacturing are very important. Man, machines and materials are called the "3Ms," or three major factors, of plants. Of these factors, a variety of inspections have been made for machines and materials. However, it was rather difficult to monitor human movements. So Daicel considered that taking images of human movements, which they could not see before, and analyzing all the factors of the 3Ms should further secure product quality. We thought it is a very new, interesting subject.
TERADABased on the position data of joints, such as hands, elbows and shoulders, of the workers obtained through the range cameras, the system statistically compares the actual movements of the workers with the standard behavioral models, which serve as the criteria. Specifically, the operations conducted by the workers are divided into small work units. Then detection is made to find if there is any movement that is different from the standard during a period of 5 to 10 seconds from the beginning to the end of each work unit. Suppose that, for example, workers fail to watch what is to be watched in the work of visual checks, or seem to "bend over" during operations that they should do while standing. Such movements are determined to be "deviations in worker activities."
Figure 2: Block diagram of detecting deviations in worker activities
NAGAYOSHIIn determining the deviations in worker activities, the standard behavioral models for respective processes play an important role. To prepare standard behavioral models, the system has to repeatedly capture the correct movements of workers on the frontlines. In doing so, the more it captures such movements, the better models it can create. On the other hand, if too many movements are captured, the data becomes too heavy to process. Therefore, we added pre-processing of smoothing to reduce noises in the joint information, and normalizing individual differences, such as the body size. This allowed us to prepare good models using fewer data.
NAGAYOSHIThat's right. Besides, the actual movements of workers have much information that is not related to their operations. For example, they do not use their legs in their operations. So collecting data of their legs has little significance. We thoroughly interviewed Daicel, the customer, as to which movements are important in relevant work operations. This helped us to reduce redundant data while keeping the number of samples small. By concentrating on the information related to work operations, we were able to create good standard behavioral models by using a limited number of samples.
NAGAYOSHIExactly. We made a lot of interviews for that purpose. In order to understand "what types of deviations in worker activities are found on the frontlines," we followed the work procedure manuals of the customer step by step, and clarified the entire work processes. While the customer knows well what operations they are conducting, we don't. To fill the gap, we patiently conducted interviews in an effort to understand the points and significance of each process.
NAGAYOSHIIn working, we were conscious of keeping speediness, expectations and interest alive. As a technique to keep the customer's expectations, we used illustrations to show them a hypothesis of what changes the system would bring to the production floors. We visually explained the advantage for the customer in an easy-to-understand manner by using images and illustrations of the analysis results, even though they are yet to obtain results. I believe this proved effective.
TERADAAt the upstream end of development, I focused on rotating the cycle of creating a prototype, making a proposal and verifying the results as quickly as possible. I had the customer verify that the system worked first by using the database built at our laboratory and then by using the data of the actual workplace.
At first, we prepared an experiment environment similar to the actual workplace at the laboratory and obtained data. At this stage, the customer was very cooperative. Their workers came to our laboratory, wore their working clothes and conducted actual operations. If we did it ourselves, our movements would have been clumsy. For that matter, we were able to obtain data that was close to the actual data of the frontlines, and verify it at the stage of preparing prototypes. We appreciated their help very much.
NAGAYOSHIThe customer evaluated our achievements, saying that the both companies offered respective technologies and data without stint and that it took only half a year to realize the system that would otherwise have required three years.
NAGAYOSHITo secure speediness, manpower is important after all. In order to respond to the requests from the customer, we let knowledgeable researchers involved in the development one after another. It also helped us greatly that they were people who can act by themselves. Even though their missions were not described in detail, the team members understood the customer's thinking by themselves and made responses on their own. That's why we were able to implement the project speedily.
TERADAWe had to use our brains in directly responding to the issues of the customer, and that was good. Usually, our business divisions or marketing people first understand the customers' issues and ask us to create certain products to address the issues. In contrast, we were able to think together with the customer from the initial stage.
On the other hand, I feel that we need to accumulate technologies for us to propose new technologies, not just responding to the customer needs.
NAGAYOSHII agree. We also have to make proper preparations for the next step. Although the customers do not tell us specifically, we need to guess what their next issues are, and conduct technological development based on certain hypotheses. Moreover, there are matters that always pose issues. For example, "how to take images of human movements accurately" remains an issue to tackle both in the security area and the industrial sector, and is a subject on which we constantly implement research, I believe.
TERADAResponding to the customers' requests and continuing cutting-edge research are what we have to do properly. I want to respond to the customers with devotion, so that I can win trust from them.
NAGAYOSHIThere is an idea of registering the technology to Lumada as a use case. We plan to generalize individual technologies, and enhance them so that they can be used proactively by other customers.
Moreover, while we have achieved risk aversion in terms of human movements and training of correct movements at the industrial sector, we want to expand the application to other areas. For that purpose, I hope to combine image analysis and artificial intelligence (AI) technologies going forward to refine the technology so that it should be able to assist human operations or propose how to work more efficiently.
TERADAPersonally, I am interested in research related to humans. I hope I can extract from videos and images what you cannot understand by directly seeing them with your eyes. It is often talked about that machines read the sentiment of persons or guess their character from their facial expressions, but that is only possible when both the images and the background data are fully obtained. Similarly, if we have such background data, we might be able to have machines read the inner aspects or thinking of people from their behaviors, I think. You cannot tell at a glance the inner aspects of people, which cause their behaviors. But if we can analyze such inner aspects from their images, we would be able to create good technologies that might be applied to a wide range of areas like security, other than the industrial sector.
NAGAYOSHIMy goal is from the perspective of promoting co-creation with customers. Recently, there is an increasing number of co-creation cases with various customers. I hope we can connect such cases. Customers with whom we work on co-creation would conduct co-creation with other customers. Suppose we work on initiatives for certain customers, and another customer would ask us to tell them about the initiatives, and we would introduce the case to the customer. Then the customer might conversely propose to us on utilizing the technologies in a different way on its own. If a technology is introduced between our customers and that generates new proposals, the future could be rosy.
