25 September 2019
By Ravneet Kaur
R&D Division, Hitachi America, Ltd.
The advent of Internet of Things (IoT) has seen fundamental changes in data acquisition, storage and analysis to create new value in multiple verticals such as manufacturing, energy, oil, gas, transportation and so on. IoT in conjunction with machine learning and AI has the potential of transforming society and improving human life. The challenges and constraints of a successful IoT deployment is also not unique but variable. A useful categorization is to contrast between IoT solutions for new greenfield applications (such as smart cities, smart home, autonomous driving) versus legacy brownfield applications (such as manufacturing, oil and gas). The latter is often termed as Industrial IoT (IIoT) [1] to distinguish from the former and typically relate to critical societal infrastructure deployments which must be in continuous 24x7 operation. Any introduction of IoT for these use cases must respect existing reliability, availability and safety constraints before attempting to improve aspects of existing operations [2].
For many such legacy societal infrastructure deployments, which is also called Operational Technology (OT), data acquisition and storage in an Information Technology (IT) database can pose a huge challenge. This could be due to legacy OT protocols for data acquisition which were designed for in-situ data visualization and manual recording. Security concerns may also limit direct physical access and networking to these controllers for data acquisition. However, having access to data in an IT database is the pre-requisite for building a successful AI solution. Thus, non-invasive data extraction methods that do not need physical connection to the existing OT networks are of great practical importance. Such methods ensure that the reliability, availability and safety constraints of the OT networks are respected while, the OT data are connected to an AI engine.
We propose such a non-invasive data extraction solution based on video analytics. The first step of our solution uses a video camera to capture images of the digital display units attached to the Programmable Logic Controller (PLC) that controls the OT assets. It then runs an edge analytics algorithm at a connected IoT gateway to automatically extract and store the values being displayed on the screen. The high-level overview of this solution is shown in Fig. 1, where the camera, gateway and analytics engine are connected to an IT network which is separate from the OT network which connects PLCs and SCADA (Supervisory Control and Data Acquisition) server.
Fig. 1. High-level overview of non-invasive data extraction
Our solution uses Optical Character Recognition (OCR) to interpret the digits on the screen. However, simply using an OCR algorithm doesn't guarantee a reliable solution due to the complexity and variability in digital display units found in factories. Our solution augments the OCR algorithm with additional modules as shown in Fig. 2.
Fig. 2. Functional blocks in proposed non-invasive data extraction solution
Image Pre-Processing: We provide the appropriate stakeholder in the factory with a tablet running a web application. The application allows its user to view the list of available machines and select one for which non-invasive data extraction is desired. A static snapshot of the controller screen for that machine appears on the application and the user can annotate the Regions of Interest (RoI) on the screen where the time-varying data is located. The coordinates of the RoI is sent to an edge analytics module that resides in an IoT gateway. This is a one-time operation. The snapshot of the web application is shown in Fig. 3.
Fig. 3. Screenshot of the web application with an image of a PLC
OCR Module: OCR is an algorithm that can recognize text in images. The edge analytics module in the IoT gateway runs an OCR algorithm to identify the characters in a user selected RoI. Thus, the OCR doesn't have to be performed on the whole complex display screen and the performance is enhanced.
Data Post-Processing: We use the domain specific context of data to catch and sometimes correct errors in the time series output from the OCR module. An example of context is if the data is a counter value, then it should be increasing till a point and then get reset to zero. Thus, if the OCR output generates a time series which is not consistent with this ‘sawtooth’ like behavior, the post-processing module flags an error. The OCR output is stored in the edge analytics module for visualization in the web application (see Fig. 4).
Fig. 4. Screenshots of the web application with graphical representation of the OCR for the selected RoI
Our aim is to operationalize the complete solution that will work for a wide range of manufacturing environments and not necessarily optimize the OCR algorithm performance for a given environment. Accordingly, we have evaluated our proposed solution under realistic manufacturing environment conditions for two OCR software - an open source software (OSS) [3] and a proprietary commercial software [4]. The accuracy of the two OCR software for the PLC displays were calculated by taking a ratio of the number of correctly recognized characters to the total number of actual characters and the results are shown in Fig. 5. None of the OCR software were specifically trained for optimized performance. We see that the overall accuracy results are high and thus, will generate a good data set for subsequent analytics applications.
Fig. 5. Accuracy comparison between the open source software and the proprietary commercial software for the data from different PLC displays
To summarize, we have proposed a solution to automatically extract information (meta-data and variable time series data values) from digital displays found in many environments such as factories. The extraction is non-invasive and thus, is very easy to deploy and possess no security risks associated with traditional methods of data extraction which require connection to the machine. We have tested our solution with realistic PLC display units used in manufacturing, under various real-world environment conditions and obtained reasonably good performance.
For more details, refer to, Non-Invasive Data Extraction from Machine Display Units Using Video Analytics, which was published in World Academy of Science - International Journal of Computer and Information Engineering 2019, 13(7), 427 – 432.
References
