Maintaining ease-of-use of on-premise systems while accelerating AI development
December 14, 2020
Hitachi, Ltd. together with the National Institute of Advanced Industrial Science and Technology (AIST), have established a hybrid*1 cloud*2 AI*3 development environment that securely connects two different information systems, Hitachi’s on-premise IT environment*4 and AIST’s ABCI cloud environment.*5 The AI development environment which begins operation this December, is equipped with job data management technology based on cloud object storage,*6 which has advanced authentication and security functions that allow data to be shared without loss of security. Further, using meta-job script*7 technology currently being developed will enable job execution*8 to be conducted with the same job scripts*9 as that used in a hybrid cloud environment, thereby providing the same usability as an on-premise environment. Hitachi and AIST plan to apply this technology to other systems such as public clouds, and by making it available to customers, contribute to the business creation for customers and the resolution of societal issues.
A hybrid cloud AI development environment
In recent years, there is an increasing use of cyber-physical systems to optimize plans such as in production, by analyzing and predicting real-world data from manufacturing equipment, automobiles, movement of people, and so on. This optimization and prediction is performed using AI, and is often implemented using hybrid cloud technology as the development of this AI requires the investment needed for advancing computing technology as well flexible response to situations such as a temporary increase in computational demand. To build a hybrid cloud AI environment, however, issues such as the need to ensure data security for confidential information and the inconvenience of dealing with systems with differing hardware and software specifications, existed. To address these issues, Hitachi together with AIST, established a hybrid cloud AI development environment that securely connects different IT systems established on-premise and on the cloud, and enables the ease-of-use of on-premise systems together with the large-scale computing resource of cloud systems.
Specifically, we developed technology using cloud object storage with advanced authentication and security functions to facilitate job data management for user authentication and data sharing between systems to address data security issues for cooperation between on-premise and cloud-based systems. Since this management technology can be run with user privileges, it does not need to be executed by a system administrator or with system administrator permission, and thus can be easily installed in a hybrid cloud. Further, using container technology that is capable of packing and transporting the libraries necessary for program execution, allows the same program to run on different information systems. The meta-job script technology used in this environment allows users to run jobs with the same job script by automatically generating job scripts for the target systems, which provides the same level of convenience as on-premise systems.
A hybrid cloud AI development environment was built by connecting the on-premise AI development environment at the Central Research Laboratory of Hitachi, Ltd. with AIST’s ABCI cloud system via SINET5*10 at the National Institute of Informatics. We then measured the response speeds for AI development jobs entered. The response speed was 0.3 seconds for jobs submitted by directly logging into ABCI which is located on the cloud, and 0.59 seconds for jobs submitted remotely from the on-premise environment, confirming that links could be established with a level of latency that does not affect user operations.
Hitachi and AIST plan to apply this technology to other systems such as public clouds, and by making it available to customers, contribute to the business creation for customers and the resolution of societal issues.
A part of the results from this project will be presented at the 177th Meeting of the High Performance Computing (HPC) Special Interest Group of the Information Procession Society of Japan (IPSJ) which will be held from 21st December 2020.
For more information, use the enquiry form below to contact the Research & Development Group, Hitachi, Ltd. Please make sure to include the title of the article.
