Joint field trial aims to build a data usability platform

Based on a consortium for collaboration between academia and industry established in April 2016, the field trial will be conducted from November 2016 through to December 2017, and carried out between the data centres in Osaka University's Cybermedia Centre and the data centres owned by Paris-based AntemetA.

In this field trial, the partners will build and test the effectiveness of a storage platform coordinated across remote locations, which offers improvements in throughput, disaster resilience, and lower costs by eliminating redundancy within individual data centres. This is done by duplicating data that is not updated very often, known as 'cold data', in data centres on each continent, and applying a data sharing technology from Scality called Conflict-free Replicated Data Type (CRDT), which is used in distributed computing environments.

Background
Currently, in order to coordinate data between data centres, two methods are used: one in which coordination is done through synchronous communications over short distances where data transfer delays are small, and another is to make backups through asynchronous communication over long distances. When using synchronous communication over short distances, there is a high probability that multiple data centres might be simultaneously affected by large-scale disasters. When asynchronous communication is used over long distances, there are large delays in transmitting data between data centres, so the only option was standby redundancy coordination for backups or disaster recovery.

At the same time, against the backdrop of changes such as the spread of technologies like AI and the IoT, data which is generally not updated, called cold data, has taken up more than half of all data, primarily in the form of image and video data, and there is an increasing demand for ways to accumulate new data and share it between multiple locations. In order to achieve data centre coordination between multiple locations, however, there have been issues in achieving data synchronisation between data centres, and in improving response speed when reading the data.

Summary of the field trial

1. Trial period
November 2016 - December 2017

2. Field trial
Details In this trial, the participants will develop the 'Geo Replication' system to realise the layering and data redundancy, which had only been achieved between data centres in close proximity or within a stand-alone data centre, at long distances such as between continents. By layering and storing not only continually updated 'hot' data and infrequently updated 'warm' data, but also 'cold' data, which is seldom updated, across multiple types of storage devices, long-term storage functionality, lowered costs, and increased responsiveness and overall throughput can be achieved across the system as a whole.

The companies participating in this consortium will advance the development of the Geo Replication system, and the field trial will be carried out, aiming for commercialisation, with partners in Europe, using data that is actually in use in Osaka University's data centres. Academic and industrial consortium participants implementing this trial (alphabetical order):

• Acutus Software
• Chuo Electronics
• Commissariat à l'énergie atomique et aux énergies alternatives
• Engineering Ingegneria Informatica S.p.A.
• FESTIVAL Project
• French Institute for Research in Computer Science and Automation
• Fujitsu
• Institut Pasteur
• Osaka University
• Panasonic
• Raritan Japan
• Scality
• Yahoo Japan Corporation

3. Division of roles

1. Osaka University:
   Responsible for controlling data layering between cold storage and the Scality Ring storage software, which is used to manage massive volumes of data 
   Technology for determining hot and cold data
   Building the Osaka site (in the Osaka University Cybermedia Centre)
2. Scality: 
   Responsible for data replication between storage functionality through Scality Ring and remote sites
   Providing Scality Ring
   Developing and providing replication functionality for this operational trial
   Building the site in France (data centres owned by AntemetA outside Paris)
3. Fujitsu:
   Responsible for managing the whole project, for implementing hot and cold storage, and for a high-speed search system for archive data
   Providing hardware such as the FUJITSU Storage ETERNUS DA700, a form of cold data storage, high performance servers like FUJITSU Server PRIMERGY RX2530, and high functionality switches
   Building the FUJITSU Intelligent Data Service SmartContentsManager high-speed archive data search system

Future developments
By the end of March 2017, the three organisations will confirm the basic operation of the system, and publish a paper on the research results from this trial during fiscal 2017. In addition, from April 2017 through December 2017, together with joint field trial partners in Japan and Europe, they will perform a usage trial for specific applications, with a variety of data types.

Based on the trial's research results, the three organisations will each aim to develop cold data storage services, a storage platform coordinated between remote locations, and other practical technologies.

Endorsements
Toshinori Kishi, Director of the Storage Division, AVC Networks Company said: "Panasonic Corporation Panasonic Corporation is participating in the Geo Replication consortium, which offers a new way of using the ETERNUS DA700, a Panasonic OEM product for Fujitsu Limited, and is very pleased to see this field trial move forward. Looking at the global market, Panasonic's data archive system using optical discs has been used by the largest social network company in the US, demonstrating that it is already becoming a realistic option for companies and public sector institutions that need long-term data storage. By adding a new way of using it for remote coordination among data centres, we expect it will also become an option in fields such as academic research and storage services.

Marc Shapiro, PhD, Senior Researcher of the French Institute for Research in Computer Science and Automation commented: "I welcome this proof-of-concept announcement, which builds upon collaboration between academia and industry, in France and Japan. The growing data needs of modern society carry the fundamental challenge of scaling both in volume and geographically. CRDTs represent a technology base for sharing data consistently across long distances in a way that is both economical and understandable to users and developers. I trust that this technology will contribute to the success of this PoC."