Railway metaverse - changing how we perceive vehicle design, maintenance and service

Two young men—one specializing in information security and another in condensed matter physics—met up in the railway metaverse

Imamoto：Making things and thinking mathematically suit me, so in senior high school I took the science track and then I went on to the electrical and information engineering department in the engineering faculty at university, where I ended up in my fourth and final year attached to the information security research group. In the first year of my master’s program, I had the opportunity to give a presentation at an international academic conference, and after that, about once a month or so, I gained experience in conference presentations. It was from around then that I started to see my future self as most likely being a researcher, so I continued doing research in information security and completed a PhD.

There was a graduate doctor in the research group who was working for Hitachi, and I first became interested in the company as a result of hearing his stories. After that, I had the chance to listen to a talk by a person who was in the department that was researching mobility control in railways and motor vehicles and eventually I decided to join Hitachi. I’d worked in research in information security the whole time, but back then I felt strongly that I wanted to make a change. I wanted to try something new, and as I was particularly interested in overall control of big systems, I was lucky to be involved from the moment I joined the company in railway mobility control.

Fujiwara：Compared to Dr. Imamoto, I spent a lot of time running around trying to find myself. Up until senior high school, my preference was for Japanese Language, but I thought it would be easier to change from science to humanities than it would be to change from humanities to science, so I tentatively chose science. I still didn’t really have a clear direction when I went to university, but when I was attached in my fourth year to a research group I encountered superconductivity for the first time, which really excited me, so I finally settled down (laughs) and chose to research condensed matter physics. My professor in the research group encouraged me to go on to postgraduate studies and I researched condensed matter physics through to my master’s.

I always liked home appliances like televisions and gaming consoles, and also it was just at the time that television was changing to digital broadcasting, so I became interested in Hitachi and joined the company. I said I wanted to research and commercialize condensed matter physics, but for some reason I was attached to the software department. Let’s face it, there’s no doubt that is what made me who I am now.

Two men—one involved in nothing but railways and another in all sorts of things—joined hands to combine railways and the metaverse

Imamoto：The whole time from when I joined the company I was involved in research having to do with railways. I worked on wireless for railways, on wireless-enabled train control systems (communication-based train control: CBTC), and on developing software to simulate propagation conditions for the purpose of confirming the safety of wireless communications. After that, I got involved in development and verification of safety functionality in rail security systems, and I’m proud to have contributed to Hitachi gaining certification for conformance with international standards. Security and safety are different concepts, but their fundamentals are the same.

Then in 2022, a project got underway to create a facility for ensuring rail safety in which it would be possible to experience manufacturing and maintenance processes in a metaverse. As part of Lumada, which is a solution that Hitachi has developed to speed up digital innovation, the initiative was to embed and leverage a range of information in a metaverse. And that’s how I came to be working with Dr. Fujiwara.

Fujiwara：That’s right. Up until then I had been working on a range of different projects involving things like automation of digital television software testing, making warehouse logistics operators more efficient, training in the maintenance of various sorts of equipment, and improving work efficiencies on site. In 2013, I encountered virtual reality (VR)—I touched it for the first time—and found it interesting and so I started teaching myself computer graphics (CG) and VR. In pursuing those ongoing external activities, in 2016, I was given a global award by Microsoft; the Microsoft Most Valuable Professional (MVP) that is given to leaders in technology communities. At the same time, the idea of making work tasks more efficient using augmented reality (AR) glasses, and my interests came together in the logistics project I was running, in the form of a project proposal, which had the fortune to be reported in a TV news program. The more I was getting involved in those activities, the more the mood, both inside and outside the company, became “if it’s AR or VR, you should talk to Fujiwara,” and so I got called in to join this railway metaverse project and that’s how I joined up with Dr. Imamoto.

Developing a metaverse for railway service using generative AI

Imamoto：Now I’m researching the application of metaverses to railways. In railways, a metaverse is likely to be used in a wide range of areas, from simulation of rail track work and service, detecting abnormalities, simulating vehicle assembly and maintenance, and feedback on design and manufacturing, to improving issuing of instructions in response to transport obstacles and designing and maintaining comfortable train station spaces.

At the R&D Group’s Ibaraki site, we used five projectors to create a metaverse facility, where humans can enter into the space and experience an environment. Here, we’re replicating the railway metaverse and the vehicle metaverse. You can experience forwards, up, down, left and right, without using HMD.

One of them, the railway metaverse, has been made by taking lots of photos in a real railway environment and composing a 3D model from the photos. it’s just a 3D model in itself, but it’s displayed in a metaverse space and overlaid with displays of operational and maintenance information.

One of the use cases for this metaverse is rail track service. We display an image of the tracks in the metaverse, and you can see how the track section has been overlayed with red and green displays. The red shows where there may be abnormalities in the track, the green is where everything’s fine. From the rail track image, we’re using AI to check for abnormalities.

Fujiwara：We’re using an image-generating AI to identify abnormalities. First, we get it to generate a “normal” image of a track. When an image is input the AI changes it to a normal track image, because it’s learned from a lot of normal track images. If an image caught with a camera at the front of a vehicle is input to the generative AI, it will output a normal track image, whether things are normal or not. When there’s an abnormality in the input image, comparing the input and output images reveals a big difference. When that happens, the display is overlayed in red.

There are all sorts of abnormal situations relating to rail service, including track abnormalities, or cracks in platforms and other facilities, for example, or fires along a line. But it’s difficult to get your hands on a lot of images of abnormalities, so getting the AI to “learn abnormalities” is unrealistic. Conversely, there are lots of images of normal tracks and platforms and lines, so what’s effective is to create an AI that generates a normal image, and then take in the difference.

Imamoto：That’s right. So then what happens is that an AI that identifies the type of abnormality kicks in. There too we use a generative AI that creates a specified abnormal situation once an image of the normal situation is input. We make it learn images of breaks in rails, cracks in platforms, and fires along lines and when images of places it’s been decided are abnormal are input, the AI generates images of breaks, cracks, and fires. It then compares those with the input image and identifies what type of abnormality is present. I said a moment ago that there’s not a lot of data on abnormalities, but there are lots of images of broken steel structures other than rail tracks, and cracks in the concrete of buildings other than train platforms, and of fires. By having the AI learn these types of images, we’ve developed technology to cause an AI to generate images of abnormal situations from input of rail tracks and railway service.

We see three major uses for the rail track metaverse. One is as a means for displaying an AR-type metaverse space at a site using smart glass or a tablet, and when a train track shows up red it will support the case for service work to be done. The second is in fully remote maintenance. It will be a means for maintenance workers working from home to look at the image of an abnormal situation overlayed with an image taken on site and to formulate a maintenance plan. The third is in training. In a metaverse you can look at the image from site of a spot requiring a repair, displayed in red, and learn in what circumstances repairs are needed. You could also simultaneously display in the metaverse space photographs and videos of maintenance methods, so trainees can learn how to do maintenance right then and there. The other thing currently being debated is developing a training kit in which generative AI deliberately creates abnormalities on rail tracks in a metaverse and new repairers have to see how many they can find. So trainees learn while having fun doing what feels like playing a computer game.

It’s also possible, of course, to replicate night scenes. Service work on railways is typically undertaken from late at night to early dawn when trains aren’t running, so it’s really important to have had experience working on site at night. Crew can learn that it really is too dark at night to see, and learn what hidden dangers there are when doing maintenance at night.

Fujiwara：A metaverse can also be replicated with HMD. HMD works if it’s a metaverse for going into a single space at different locations, but if you want to be in the same physical space and share multiple bits of information as you check conditions, then a realistic space-type metaverse created with projectors is appropriate. You could say that what we’re doing is kind of synthesizing a metaverse space with real people’s conversations.

More applications in vehicle fit out and design and maintenance

Imamoto：We’re also developing a metaverse just for vehicles. What we’re demonstrating here is a vehicle metaverse created using actual vehicle CAD data. We anticipate that the metaverse inside the vehicle will assist in fitting out vehicles and design and maintenance. First, we’re displaying in the metaverse the situation when you’re just riding in an ordinary vehicle, but we can blank out the seats and so on and just look at the shell, or look at the bogie from inside the vehicle, or show the display from the under car, which is the actual maintenance perspective. Floating in this vehicle metaverse there are a lot of what look to be lamps. They’re tethered to vehicle operations information and information about design specification changes. So you don’t just look at information about design changes, you can actually vicariously experience in the metaverse the situation that made it necessary to change the design, by displaying a comparison of the specifications before and after the design change.

Fujiwara：It’s not that difficult to display scenery of train tracks or vehicle spaces as a metaverse, but what’s important is how to incorporate the appropriate information into it and whether or not it can be used as a basis for service and design and maintenance. We think that if everyone is able to go into a metaverse and easily share information, it should, for example, make it easier for people who are coming from different standpoints to get onto the same page and achieve consensus. The theme of research ongoing is how to provide information that will lead to even greater depths of understanding. And at this stage, we absolutely need the knowledge of people who are familiar with the situation on site, so the project is being advanced in very close association with the operations division.