Performance characterization of spark workloads on shared NUMA Systems
Document typeConference report
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Rights accessOpen Access
European Commission's projectHi-EST - Holistic Integration of Emerging Supercomputing Technologies (EC-H2020-639595)
As the adoption of Big Data technologies becomes the norm in an increasing number of scenarios, there is also a growing need to optimize them for modern processors. Spark has gained momentum over the last few years among companies looking for high performance solutions that can scale out across different cluster sizes. At the same time, modern processors can be connected to large amounts of physical memory, in the range of up to few terabytes. This opens an enormous range of opportunities for runtimes and applications that aim to improve their performance by leveraging low latencies and high bandwidth provided by RAM. The result is that there are several examples today of applications that have started pushing the in-memory computing paradigm to accelerate tasks. To deliver such a large physical memory capacity, hardware vendors have leveraged Non-Uniform Memory Architectures (NUMA). This paper explores how Spark-based workloads are impacted by the effects of NUMA-placement decisions, how different Spark configurations result in changes in delivered performance, how the characteristics of the applications can be used to predict workload collocation conflicts, and how to improve performance by collocating workloads in scale-up nodes. We explore several workloads run on top of the IBM Power8 processor, and provide manual strategies that can leverage performance improvements up to 40% on Spark workloads when using smart processor-pinning and workload collocation strategies.
CitationBaig, S., Amaral, M., Polo, J., Carrera, D. Performance characterization of spark workloads on shared NUMA Systems. A: International Conference on Big Data Computing Service and Applications. "2018 IEEE Fourth International Conference on Big Data Computing Service and Applications (BigDataService 2018): Bamberg, Germany: 26-29 March 2018". Institute of Electrical and Electronics Engineers (IEEE), 2018, p. 41-48.
- CAP - Grup de Computació d'Altes Prestacions - Ponències/Comunicacions de congressos 
- Departament d'Arquitectura de Computadors - Ponències/Comunicacions de congressos [1.635]
- Departament d'Enginyeria Electrònica - Ponències/Comunicacions de congressos [1.490]
- GRUP ISI - Grup d'Instrumentació, sensors i interfícies - Ponències/Comunicacions de congressos 
All rights reserved. This work is protected by the corresponding intellectual and industrial property rights. Without prejudice to any existing legal exemptions, reproduction, distribution, public communication or transformation of this work are prohibited without permission of the copyright holder