Configuring parallelism for hybrid layouts using multi-objective optimization
Visualitza/Obre
Cita com:
hdl:2117/336927
Tipus de documentArticle
Data publicació2020-06-01
Condicions d'accésAccés obert
Tots els drets reservats. Aquesta obra està protegida pels drets de propietat intel·lectual i
industrial corresponents. Sense perjudici de les exempcions legals existents, queda prohibida la seva
reproducció, distribució, comunicació pública o transformació sense l'autorització del titular dels drets
Abstract
Modern organizations typically store their data in a raw format in data lakes. These data are then processed and usually stored under hybrid layouts, because they allow projection and selection operations. Thus, they allow (when required) to read less data from the disk. However, this is not very well exploited by distributed processing frameworks (e.g., Hadoop, Spark) when analytical queries are posed. These frameworks divide the data into multiple partitions and then process each partition in a separate task, consequently creating tasks based on the total file size and not the actual size of the data to be read. This typically leads to launching more tasks than needed, which, in turn, increases the query execution time and induces significant waste of computing resources. To allow a more efficient use of resources and reduce the query execution time, we propose a method that decides the number of tasks based on the data being read. To this end, we first propose a cost-based model for estimating the size of data read in hybrid layouts. Next, we use the estimated reading size in a multi-objective optimization method to decide the number of tasks and computational resources to be used. We prototyped our solution for Apache Parquet and Spark and found that our estimations are highly correlated (0.96) with the real executions. Further, using TPC-H we show that our recommended configurations are only 5.6% away from the Pareto front and provide 2.1 × speedup compared with default solutions.
CitacióMunir, R. [et al.]. Configuring parallelism for hybrid layouts using multi-objective optimization. "Big data", 1 Juny 2020, vol. 8, núm. 3, p. 235-247.
ISSN2167-6461
Versió de l'editorhttps://www.liebertpub.com/doi/10.1089/big.2019.0068
Col·leccions
- Doctorat Erasmus Mundus en Tecnologies de la Informació per a la Intel·ligència Empresarial - Articles de revista [6]
- inSSIDE - integrated Software, Service, Information and Data Engineering - Articles de revista [113]
- Departament d'Enginyeria de Serveis i Sistemes d'Informació - Articles de revista [222]
- IMP - Information Modeling and Processing - Articles de revista [126]
Fitxers | Descripció | Mida | Format | Visualitza |
---|---|---|---|---|
bigdata2020.pdf | 881,2Kb | Visualitza/Obre |