As companies handle an increasingly larger volume of data, all flash arrays that feature large memory capacity and fast processing speed are becoming progressively popular. Hitachi, Ltd. has developed all flash arrays, called Hitach Virtual Storage Platform F Series, which incorporate data compression technologies. To develop them, the Company's researchers from a variety of technical areas have successfully met the challenge of reducing the bit cost of all flash arrays through data compression, while keeping them performing at a high level.
(Publication: October 14, 2016)
MATSUSHITAYes, that's right. Looking ahead, we will see an expansion of IoT (Internet of Things) businesses, in which a large volume of sensor data is analyzed and the findings are provided to customers as a new value. Because of this and other reasons, it is projected that the data volume handled by companies will explosively expand going forward. To keep up with such trends of society, they will require storage devices featuring a large memory capacity and high processing speed. Our customers who require such high-performance storage devices are quickly switching to all flash arrays, which incorporate flash drives like solid state drives (SSDs) as memory devices.
KOSEKIIn the case of an all flash array, the time from when it is given an instruction to read certain data it stores to when it gives back the relevant data is far shorter than a conventional storage device incorporating a hard disk drive (HDD). For example, when you make an on-line reservation from a website, you want to receive a reply immediately after you click the reservation button, don't you? Similarly, for customers who want to finish processing as soon as possible, the high-performance all flash arrays are a very attractive alternative.
KOSEKIIn 2012, the Company announced the Hitachi Accelerated Flash (HAF), a flash drive module (hereafter, "flash module") of its own. Since then, as other companies have announced many flash drives, the price has been falling remarkably. However, it is true that flash drives, or all flash arrays, are still more expensive than HDDs, or HDD arrays. Thus, as a means to further reduce the bit cost (price per memory capacity), data compression technologies are gathering attention.
MATSUSHITAFor example, if you write 100 kilobytes (KB) of data onto a storage device, you usually require a memory capacity of 100 KB. However, if you can cut the size in half by compressing the data, you only need to buy memory capacity of 50 KB. In this case, the price per memory capacity is halved, which helps to lower the price of the storage device.
If the data compression technologies can change the general perception that "all flash arrays are expensive," it will facilitate further popularization and widespread use of all flash arrays.
MATSUSHITA I agree. But data compression technologies have had the major issue that they lower the input/output (I/O) performance of the all flash arrays.
Generally speaking, compression processing and decompression processing, which reverses compressed data to a non-compressed state, is conducted via software that operates with the storage controller. On the other hand, I/O processing (processing for reading and writing data) is also conducted by the same storage controller. Therefore, the compression and decompression processing tasks slow the I/O processing speed.
The purpose of using all flash arrays is to finish I/O processing as quickly as possible. If the I/O processing speed is slowed because of the compression and decompression processing tasks, it is like putting the cart before the horse, isn't it? So the big challenge is to find how to conduct data compression while maintaining the I/O performance of the all flash arrays.
KOSEKITo meet the challenge, we devised a method to diversify the compression and decompression processing tasks into multiple flash modules, as shown in Figure 1.
In an all flash array, a lot of flash modules are connected to the storage controller with which the software operates. We thought that, by having the flash modules share the compression and decompression processing tasks, the storage controller should have less burden so that the all flash array would maintain its I/O performance.
Figure1: Difference in how data is compressed
KOSEKIOur newly developed technology has two main points. The first is that it enables flash modules to perform compression and decompression processing tasks at a high speed.
In this regard, we have newly developed a special flash module called the Hitachi Accelerated Flash Data Compression 2 (HAF DC2), which is the second-generation product of HAF.
The HAF DC2 incorporates a semiconductor-based compression engine, which performs compression and decompression of data at a high speed. In addition, the compression engine features Hitachi's original algorithm, which works to maximize the processing performance of the device.
MATSUSHITAThe second is how to manage the memory capacity of the flash modules. We had to solve the issue of how we allow the storage controller to use the free memory space that has become available as a result of data compression.
Suppose a flash module has a memory capacity of 100 KB. For this flash module, the storage controller writes data of 100 KB, and the compression engine of the flash module halves the data size so that the data compressed to 50 KB is stored. In this case, the storage controller, which has written 100 KB data, considers that the flash module has no free space in its capacity (free capacity) any more. In other words, even if the flash module has free capacity as a result of data compression, the storage controller cannot write more data there unless it recognizes the existence of the free capacity.
In order to address this issue, the new flash modules first provide the storage controller with a virtual memory capacity that is larger than what can be written physically. On the other hand, the storage controller monitors how much space is left in the memory capacity of the flash modules. Moreover, as Figure 2 shows, the free capacity of each flash module is combined to form an available memory capacity as a whole and is provided to the storage manager. Through these measures, the free capacity created by data compression can be utilized effectively.
This scheme has been patented.* We believe this method was made possible just because Hitachi produces both the storage controller and the flash modules on its own.
Figure 2: How the storage controller manages the free capacity of the flash modules
KOSEKIExactly. In past development efforts, there had been two different teams involved. One was engaged in developing such devices as flash modules, and the other worked to develop storage controllers. The teams independently designed the functions of respective products, and had few opportunities to directly discuss with each other.
In our latest development, however, the two teams worked together to achieve a specific function. This meant that, if either one of the teams was delayed in their development work, no evaluation of the function was possible. Furthermore, if any failure occurred, they had to arrange how to repair it and decide which team should secure the operations of the products to what degree. In order to avoid any discrepancy or inconsistency arising between the teams, we worked to assure that the teams elaborately discussed to determine the specifications.
MATSUSHITAOn the laboratory side, which we belong to, we decided on the entire framework, including which scheme to adopt. The scheme proposed by the laboratory was approved by the relevant business division. However, we faced difficulties after development started at the business division. I already mentioned that the free capacity of each flash module is monitored and combined to provide the available memory capacity to the users. Reflecting this idea into designing of the actual product was what we found difficult.
MATSUSHITAFor the persons in charge of software development in the business division, it was understood that such devices as HDDs, SSDs and flash modules have a certain fixed memory capacity which remains unchanged. In this development, however, the free capacity of the flash modules changes depending on the data compression ratio. Therefore, you must control the devices in which you have no idea how much data can be written. For the business division, this was a totally different story from what they had assumed earlier. So we worked together with the business division to verify the appropriateness of the scheme from the viewpoints of whether the devices can actually be controlled and whether it is sure that the scheme causes no problems.
An example of what we discussed was: "The storage controller monitors the free capacity of the flash modules at the frequency of once every so-and-so seconds. During the cycle, the storage controller receives up to so-and-so number of 'write' requests. Then, isn't there any possibility that the flash modules receive more data than they can contain?" We tackled such issues one by one by gathering the ideas and wisdom of the team members, and after all these endeavors we successfully completed the product.
MATSUSHITAI was involved in the project for a long time, as I had been working all the time in the business division for about a year and a half before the product launch. When the product was completed, I shared the joy with the members of the business division. I was the only researcher of the storage controller among the laboratory members who stayed full-time in the business division. Partly because of this, the business division people cared for me as if I were one of their peers, and I was able to establish a strong connection with them. This is a great treasure for me, and I want to nourish it going forward.
KOSEKIOur stance is to cooperate with the members from the business division and, instead of simply devising the scheme, work all the way to applying the scheme to the product. I feel encouraged and rewarded when I see the research results materialize in a product and that serves society. Moreover, it is easier for me to explain to my family that I am making such a product. It's unlucky, though, that I cannot show them how the product actually works, because the product is used in core systems and other areas that ordinary people cannot see.
MATSUSHITAIt seems to me that, with the widespread use of the all flash arrays, the customers now have a different view of storage devices. Conventionally, it was generally believed that "storage devices are the slower part of the entire IT system." The storage managers in the IT departments use to frequently tune them to make them perform to the best of their ability, and Hitachi's storage devices also allowed a variety of settings for tuning up their performance.
More recently, however, there is a trend in which business departments directly purchase IT products and use them as a platform to launch new businesses. Our customers from such business departments, who are not specialists of IT products, sometimes tell us that they want to reduce the set-up and tuning operations as much as possible. Because of this, and with the number of managers specialized in storage devices decreasing, research would be needed to make storage devices easier to use. Then, you would be able to use them as easily as refrigerators, washing machines and other home appliances. Such ease of use should provide value for customers.
KOSEKIWhat added value should we provided to customers so that they choose Hitachi's storage devices from among the many storage products? To answer this question, we should grasp the changes in the market and in the industry as soon as possible and propose specific ideas. I believe that this is the issue that we must tackle going forward.
MATSUSHITAWhen I was involved in supporting development in the business division, I strongly felt the importance of project management. Software development for storage devices is a very large-scale project, and promoting such a large-scale project smoothly is not a job that we can readily do. In doing either research or product development, we often need to cooperate with a variety of people. Therefore, I hope to foster and practice management skills so that I can play an active role as a team leader.
KOSEKISince I joined the Company, I have been engaged in applied research, which is closer to product development. More recently, however, I wish to have the opportunity to conduct basic research—a type of research that would plant seeds of new technologies, although they may not become practically available soon. In my view, what appeals to the researchers at Hitachi is that they can contribute to society in a variety of forms and ways.
