Accelerating Complex Data Transfer for Cluster Computing

Alexey Khrabrov, Eyal de Lara

8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16), Denver, CO, June 2016



The ability to move data quickly between the nodes of a distributed system is important for the performance of cluster computing frameworks, such as Hadoop and Spark. We show that in a cluster with modern networking technology data serialization is the main bottleneck and source of overhead in the transfer of rich data in systems based on high-level programming languages such as Java. We propose a new data transfer mechanism that avoids serialization altogether by using a shared clusterwide address space to store data. The design and a prototype implementation of this approach are described. We show that our mechanism is significantly faster than serialized data transfer, and propose a number of possible applications for it.