Keeping the data lake in form: proximity mining for pre-filtering schema matching
Rights accessOpen Access
Data Lakes (DLs) are large repositories of raw datasets from disparate sources. As more datasets are ingested into a DL, there is an increasing need for efficient techniques to profile them and to detect the relationships among their schemata, commonly known as holistic schema matching. Schema matching detects similarity between the information stored in the datasets to support information discovery and retrieval. Currently, this is computationally expensive with the volume of state-of-the-art DLs. To handle this challenge, we propose a novel early-pruning approach to improve efficiency, where we collect different types of content metadata and schema metadata about the datasets, and then use this metadata in early-pruning steps to pre-filter the schema matching comparisons. This involves computing proximities between datasets based on their metadata, discovering their relationships based on overall proximities and proposing similar dataset pairs for schema matching. We improve the effectiveness of this task by introducing a supervised mining approach for effectively detecting similar datasets which are proposed for further schema matching. We conduct extensive experiments on a real-world DL which proves the success of our approach in effectively detecting similar datasets for schema matching, with recall rates of more than 85% and efficiency improvements above 70%. We empirically show the computational cost saving in space and time by applying our approach in comparison to instance-based schema matching techniques.
CitationAl-serafi, A. [et al.]. Keeping the data lake in form: proximity mining for pre-filtering schema matching. "ACM transactions on information systems", Maig 2020, vol. 38, núm. 3, article 26, p. 1-30.
- Doctorat Erasmus Mundus en Tecnologies de la Informació per a la Intel·ligència Empresarial - Articles de revista 
- inSSIDE - integrated Software, Service, Information and Data Engineering - Articles de revista 
- Departament d'Enginyeria de Serveis i Sistemes d'Informació - Articles de revista 
- IMP - Information Modeling and Processing - Articles de revista 
All rights reserved. This work is protected by the corresponding intellectual and industrial property rights. Without prejudice to any existing legal exemptions, reproduction, distribution, public communication or transformation of this work are prohibited without permission of the copyright holder