Intermediate results materialization selection and format for data-intensive flows
Fitxers
Títol de la revista
ISSN de la revista
Títol del volum
Col·laborador
Tribunal avaluador
Realitzat a/amb
Tipus de document
Data publicació
Editor
Condicions d'accés
item.page.rightslicense
Publicacions relacionades
Datasets relacionats
Projecte CCD
Abstract
Data-intensive flows deploy a variety of complex data transformations to build information pipelines from data sources to different end users. As data are processed, these workflows generate large intermediate results, typically pipelined from one operator to the following ones. Materializing intermediate results, shared among multiple flows, brings benefits not only in terms of performance but also in resource usage and consistency. Similar ideas have been proposed in the context of data warehouses, which are studied under the materialized view selection problem. With the rise of Big Data systems, new challenges emerge due to new quality metrics captured by service level agreements which must be taken into account. Moreover, the way such results are stored must be reconsidered, as different data layouts can be used to reduce the I/O cost. In this paper, we propose a novel approach for automatic selection of multi-objective materialization of intermediate results in data-intensive flows, which can tackle multiple and conflicting quality objectives. In addition, our approach chooses the optimal storage data format for selected materialized intermediate results based on subsequent access patterns. The experimental results show that our approach provides 40% better average speedup with respect to the current state-of-the-art, as well as an improvement on disk access time of 18% as compared to fixed format solutions.
Descripció
Persones/entitats
Document relacionat
item.page.versionof
Citació
Ajut
Forma part
Dipòsit legal
ISBN
ISSN
Versió de l'editor
Altres identificadors
Referències
Col·leccions
inSSIDE - integrated Software, Service, Information and Data Engineering - Articles de revista
Departament d'Enginyeria de Serveis i Sistemes d'Informació - Articles de revista
GESSI - Grup d'Enginyeria del Software i dels Serveis - Articles de revista
IMP - Information Modeling and Processing - Articles de revista