Towards scalable data discovery

View/Open
Cita com:
hdl:2117/343141
Document typeConference lecture
Defense date2021
PublisherOpenProceedings
Rights accessOpen Access
This work is protected by the corresponding intellectual and industrial property rights.
Except where otherwise noted, its contents are licensed under a Creative Commons license
:
Attribution-NonCommercial-NoDerivs 4.0 International
Abstract
We study the problem of discovering joinable datasets at scale. We approach the problem from a learning perspective relying on profiles. These are succinct representations that capture the underlying characteristics of the schemata and data values of datasets, which can be efficiently extracted in a distributed and parallel fashion. Profiles are then compared, to predict the quality of a join operation among a pair of attributes from different datasets. In contrast to the state-of-the-art, we define a novel notion of join quality that relies on a metric considering both the containment and cardinality proportion between join candidate attributes. We implement our approach in a system called NextiaJD, and present experiments to show the predictive performance and computational efficiency of our method. Our experiments show that NextiaJD obtains similar predictive performance to that of hash-based methods, yet we are able to scale-up to larger volumes of data. Also, NextiaJD generates a considerably less amount of false positives, which is a desirable feature at scale.
CitationFlores, J.; Nadal, S.; Romero, O. Towards scalable data discovery. A: International Conference on Extending Database Technology. "Advances in Database Technology: EDBT 2021, 24th International Conference on Extending Database Technology: Nicosia, Cyprus, March 23-26, 2021: proceedings". Konstanz: OpenProceedings, 2021, p. 433-438. ISBN 978-3-89318-084-4. DOI 10.5441/002/edbt.2021.47.
ISBN978-3-89318-084-4
Publisher versionhttps://doi.org/10.5441/002/edbt.2021.47
Collections
- Doctorat en Computació - Ponències/Comunicacions de congressos [71]
- inSSIDE - integrated Software, Service, Information and Data Engineering - Ponències/Comunicacions de congressos [332]
- Departament d'Enginyeria de Serveis i Sistemes d'Informació - Ponències/Comunicacions de congressos [566]
- Doctorat Erasmus Mundus en Tecnologies de la Informació per a la Intel·ligència Empresarial - Ponències/Comunicacions de congressos [11]