Spark deployment and performance evaluation on the MareNostrum supercomputer
Document typeConference report
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Rights accessOpen Access
In this paper we present a framework to enable data-intensive Spark workloads on MareNostrum, a petascale supercomputer designed mainly for compute-intensive applications. As far as we know, this is the first attempt to investigate optimized deployment configurations of Spark on a petascale HPC setup. We detail the design of the framework and present some benchmark data to provide insights into the scalability of the system. We examine the impact of different configurations including parallelism, storage and networking alternatives, and we discuss several aspects in executing Big Data workloads on a computing system that is based on the compute-centric paradigm. Further, we derive conclusions aiming to pave the way towards systematic and optimized methodologies for fine-tuning data-intensive application on large clusters emphasizing on parallelism configurations.
CitationTous, R., Gounaris, A., Tripiana, C., Torres, J., Girona, S., Ayguadé, E., Labarta, J., Becerra, Y., Carrera, D., Valero, M. Spark deployment and performance evaluation on the MareNostrum supercomputer. A: IEEE International Conference on Big Data. "2015 IEEE International Conference on Big Data: Oct 29-Nov 01, 2015, Santa Clara, CA, USA: proceedings". Santa Clara, CA: Institute of Electrical and Electronics Engineers (IEEE), 2015, p. 299-306.