Mostra el registre d'ítem simple
DDS: integrating data analytics transformations in task-based workflows [version 1; peer review: 1 approved, 2 approved with reservations]
dc.contributor.author | Mammadli, Nihad |
dc.contributor.author | Ejarque Artigas, Jorge |
dc.contributor.author | Álvarez Cid-Fuentes, Javier |
dc.contributor.author | Badia Sala, Rosa Maria |
dc.contributor.other | Universitat Politècnica de Catalunya. Departament d'Arquitectura de Computadors |
dc.contributor.other | Barcelona Supercomputing Center |
dc.date.accessioned | 2023-01-10T10:18:32Z |
dc.date.available | 2023-01-10T10:18:32Z |
dc.date.issued | 2022-05-25 |
dc.identifier.citation | Mammadli, N. [et al.]. DDS: integrating data analytics transformations in task-based workflows [version 1; peer review: 1 approved, 2 approved with reservations]. "Open Research Europe", 25 Maig 2022, vol. 2, article 66, p. 1-16. |
dc.identifier.issn | 2732-5121 |
dc.identifier.uri | http://hdl.handle.net/2117/379643 |
dc.description.abstract | High-performance data analytics (HPDA) is a current trend in e-science research that aims to integrate traditional HPC with recent data analytic frameworks. Most of the work done in this field has focused on improving data analytic frameworks by implementing their engines on top of HPC technologies such as Message Passing Interface. However, there is a lack of integration from an application development perspective. HPC workflows have their own parallel programming models, while data analytic (DA) algorithms are mainly implemented using data transformations and executed with frameworks like Spark. Task-based programming models (TBPMs) are a very efficient approach for implementing HPC workflows. Data analytic transformations can also be decomposed as a set of tasks and implemented with a task-based programming model. In this paper, we present a methodology to develop HPDA applications on top of TBPMs that allow developers to combine HPC workflows and data analytic transformations seamlessly. A prototype of this approach has been implemented on top of the PyCOMPSs task- based programming model to validate two aspects: HPDA applications can be seamlessly developed and have better performance than Spark. We compare our results using different programs. Finally, we conclude with the idea of integrating DA into HPC applications and evaluation of our method against Spark. |
dc.description.sponsorship | This research was financially supported by the European Union’s Horizon 2020 research and innovation programme under the grant agreement No 780622; and the Spanish Government (PID2019-107255GB), Generalitat de Catalunya (2014-SGR-1051). |
dc.format.extent | 16 p. |
dc.language.iso | eng |
dc.rights | Attribution 4.0 International |
dc.rights.uri | http://creativecommons.org/licenses/by/4.0/ |
dc.subject | Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors |
dc.subject | Àrees temàtiques de la UPC::Informàtica::Programació |
dc.subject.lcsh | Big data |
dc.subject.lcsh | High performance computing |
dc.subject.lcsh | Parallel programming (Computer science) |
dc.subject.other | Big data high performance |
dc.subject.other | Data analytics |
dc.subject.other | Parallel computing |
dc.subject.other | Task based programming models |
dc.title | DDS: integrating data analytics transformations in task-based workflows [version 1; peer review: 1 approved, 2 approved with reservations] |
dc.type | Article |
dc.subject.lemac | Dades massives |
dc.subject.lemac | Càlcul intensiu (Informàtica) |
dc.subject.lemac | Programació en paral·lel (Informàtica) |
dc.identifier.doi | 10.12688/openreseurope.14569.1 |
dc.description.peerreviewed | Peer Reviewed |
dc.relation.publisherversion | https://open-research-europe.ec.europa.eu/articles/2-66/v1#referee-response-29377 |
dc.rights.access | Open Access |
local.identifier.drac | 35033561 |
dc.description.version | Postprint (published version) |
dc.relation.projectid | info:eu-repo/grantAgreement/EC/H2020/780622/EU/Edge and CLoud Computation: A Highly Distributed Software Architecture for Big Data AnalyticS/CLASS |
dc.relation.projectid | info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/PID2019-107255GB-C22/ES/UPC-COMPUTACION DE ALTAS PRESTACIONES VIII/ |
dc.relation.projectid | info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/PID2019-107255GB-C21/ES/BSC - COMPUTACION DE ALTAS PRESTACIONES VIII/ |
local.citation.author | Mammadli, N.; Ejarque, J.; Álvarez, J.; Badia, R.M. |
local.citation.publicationName | Open Research Europe |
local.citation.volume | 2 |
local.citation.number | article 66 |
local.citation.startingPage | 1 |
local.citation.endingPage | 16 |
Fitxers d'aquest items
Aquest ítem apareix a les col·leccions següents
-
Articles de revista [1.049]