wave

Transcontinental Data Transfer
Poland-Singapore data transfer over new CAE-1 100G trans-continental link

Poland- Singapore Data Transfer

In early October 2019, Interdisciplinary Centre for Mathematical and Computational Modelling (ICM) – University of Warsaw (Poland), A*STAR Computational Resource Centre (A*CRC, Singapore), and Zettar Inc. (U.S.) embarked to jointly conduct a production trial over the newly built Collaboration Asia Europe-1 (CAE-1) 100Gbps link connecting London and Singapore.

ICM presented this transcontinental link as a production ready infrastructure at SC19 in Denver, 2019.

The link provides shorter, faster, and cheaper connectivity than the links routed via the North Atlantic Ocean, across North America, and across the Pacific Ocean that have carried much of the Research&Education (R&E) traffic to date between Europe and Asia Pacific region. Furthermore, with the link, the Middle East region is now able to participate in globally distributed data-intensive research and scientific endeavors with Europe, Asia Pacific region, and beyond.

But how well does it work in practice? The 3 parties decided to find it out using entirely production grade components: hardware, storage, network infrastructure, and software.

Moving data at great speed and scale

The project has established a historical first: for the first time over the newly built CAE-1 link, with a production setup at ICM end. It has shown that moving data at great speed and scale between Poland (and thus Central and Eastern Europe) and Singapore is a reality. Furthermore, although the project was initiated only in mid-October, all goals have been reached and a few new grounds have also been broken as well. On the ICM side only two technical experts were involved: Marcin Semeniuk, who configured the entire set-up on the Polish side and Jarosław Skomiał who was responsible for establishing a data link between Warsaw and Singapore. The idea for this production environment was proposed by the former director of ICM, Dr. Marek Michalewicz, who also coordinated this project with all international collaborators.

It is also a true international collaboration:

  • ICM, aka Interdisciplinary Centre for Mathematical and Computational Modelling, University of Warsaw, Poland is one of the most established supercomputing centers in Eastern Europe;
  • A*CRC , aka A*STAR Computational Resource Centre, is the Singapore government-funded source of HPC expertise;
  • Zettar Inc. is a software startup based in Palo Alto, California, U.S. It is supported by its revenue and U.S. DOE Office of Science funding. It delivers a software application zx for moving data at speed and scale since 2016 and has been setting a world record annually ever since.

Furthermore, in the social, research and scientific collaboration, and engineering, the project has achieved many worthy accomplishments. We had the pleasure to present it during Expo 2020 Dubai. The webinar was shown at Poland Expo.

The project has shown concretely the following:

  • More R&E regions are reachable. From now on, distributed data-intensive science and engineering collaboration between Europe, Middle East, Asia and Pacific regions are not only feasible, but also can be efficient if the right data moving solution is used.
  • More world-wide participation in distributed data-intensive research collaboration is a reality. The achievement should encourage and motivate more parties along the data path and beyond to collaborate on the advancement of the global sciences and engineering.
  • Date gravity is no longer a barrier to progress. Even with the tight time for preparation, the attained transfer speed already shows it’s possible to move 1PB in less than two days between any two points along the data path used by the project.

Engineering:

  1. Modest hardware can produce world-class top results, if the resources are utilized intelligently.
  2. This is a production trial – not a “for show demo”. For example, at ICM, two production Lustre file systems are employed; both formed with 20 OSTs; each OST has 4 x 7200RPM HDDs. Not even a single SSD is employed. Only a single DTN at each end. Both DTNs are from existing hardware inventory. Both DTNs are more than 2 years old.
  3. Attained result is world’s top level (~60Gbps average)
  4. Stock TCP is used. There is no need to use any proprietary protocol.
  5. Vast distance: 19,800 km,12,375 miles
  6. Stunningly short preparation: 2 weeks total
  7. InfiniBand (IB), typically used for interconnect in the HPC space, is not amenable to interface bonding, unlike Ethernet, But the two storage pools with IB interconnects are aggregated by the data mover software Zettar zx.

Scientific team at ICM

The application and development of modern network solutions in the area of ​​big data transfer and is carried out by a scientific team at ICM, composed of: Jarosław Skomiał, (expert in network technologies and data transfer), Marcin Semeniuk (expert in network and computing technologies) and Karol Niedzielewski (expert in network technologies and scientific applications). The team was formerly led by Dr. Marek Michalewicz.

Here are some examples:

  1. Connecting two ICM data centers (Ochota – Białołęka) at a distance of approx. 20 km with a throughput of 1.2 Tbps using the latest CloudXpress-2 demonstration equipment from Infinera (ICM was the first to test this technology in Europe, right after Amazon, Facebook and Google conducted their tests).
  2. Data Transfer Nodes (DTN) – combined with data transfer and computations on a global scale. At the Supercomputing 2018 conference in the US the connection between ICM in Warsaw with the Pawsey Center in Perth, Australia and launching containerized programs alternatively either on resources in Australia or in Warsaw was demonstrated.
  3. Establishing InfiniBand connections between Polish supercomputing centers: ICM and TASK Gdańsk (about 900 km light path), and ICM and NCBJ, Świerk (about 40km light path) and building highly distributed concurrent computer systems.
  4. Data transfer between Warsaw and Singapore on the new 100Gbps CEA-1 (Collaboration Asia Europe 1) connection at 100Gbps in cooperation with US based company Zettar.

Jarosław Skomiał

EMAIL arrow
wave

POLAND PAVILION
AT EXPO 2020 DUBAI

Poland Pavilion
1 October 2021 – 31 March 2022

WHEN

From 10:00 to 22:00

WHERE

“Mobility” sector Expo Dubai 2020

Virtual Expo Dubai arrow
Skip to content