Mostra el registre d'ítem simple
A programming model for hybrid workflows: combining task-based workflows and dataflows all-in-one
dc.contributor.author | Ramón Cortés, Cristian |
dc.contributor.author | Lordan Gomis, Francesc |
dc.contributor.author | Ejarque Artigas, Jorge |
dc.contributor.author | Badia Sala, Rosa Maria |
dc.contributor.other | Universitat Politècnica de Catalunya. Doctorat en Arquitectura de Computadors |
dc.contributor.other | Universitat Politècnica de Catalunya. Departament d'Arquitectura de Computadors |
dc.contributor.other | Barcelona Supercomputing Center |
dc.date.accessioned | 2020-09-17T10:29:37Z |
dc.date.available | 2022-07-09T00:28:46Z |
dc.date.issued | 2020-12 |
dc.identifier.citation | Ramón-Cortés, C. [et al.]. A programming model for hybrid workflows: combining task-based workflows and dataflows all-in-one. "Future generation computer systems", Desembre 2020, vol. 113, p. 281-297. |
dc.identifier.issn | 0167-739X |
dc.identifier.other | https://arxiv.org/abs/2007.04939 |
dc.identifier.uri | http://hdl.handle.net/2117/328850 |
dc.description.abstract | In the past years, e-Science applications have evolved from large-scale simulations executed in a single cluster to more complex workflows where these simulations are combined with High-Performance Data Analytics (HPDA). To implement these workflows, developers are currently using different patterns; mainly task-based and dataflow. However, since these patterns are usually managed by separated frameworks, the implementation of these applications requires to combine them; considerably increasing the effort for learning, deploying, and integrating applications in the different frameworks. This paper tries to reduce this effort by proposing a way to extend task-based management systems to support continuous input and output data to enable the combination of task-based workflows and dataflows (Hybrid Workflows from now on) using a single programming model. Hence, developers can build complex Data Science workflows with different approaches depending on the requirements. To illustrate the capabilities of Hybrid Workflows, we have built a Distributed Stream Library and a fully functional prototype extending COMPSs, a mature, general-purpose, task-based, parallel programming model. The library can be easily integrated with existing task-based frameworks to provide support for dataflows. Also, it provides a homogeneous, generic, and simple representation of object and file streams in both Java and Python; enabling complex workflows to handle any data type without dealing directly with the streaming back-end. During the evaluation, we introduce four use cases to illustrate the new capabilities of Hybrid Workflows; measuring the performance benefits when processing data continuously as it is generated, when removing synchronisation points, when processing external real-time data, and when combining task-based workflows and dataflows at different levels. The users identifying these patterns in their workflows may use the presented uses cases (and their performance improvements) as a reference to update their code and benefit of the capabilities of Hybrid Workflows. Furthermore, we analyse the scalability in terms of the number of writers and readers and measure the task analysis, task scheduling, and task execution times when using objects or streams. |
dc.description.sponsorship | This work has been supported by the Spanish Government (contracts SEV2015-0493 and TIN2015-65316-P), by Generalitat de Catalunya (contract 2014-SGR-1051), and by the European Commission through the Horizon 2020 Research and Innovation program under contract 730929 (MF2C project). Cristian Ramon-Cortes predoctoral contract is financed by the Spanish Government under the contract BES-2016-076791. |
dc.format.extent | 17 p. |
dc.language.iso | eng |
dc.publisher | Elsevier |
dc.rights | Attribution-NonCommercial-NoDerivatives 4.0 International |
dc.rights | ©2020 Elsevier |
dc.rights.uri | https://creativecommons.org/licenses/by-nc-nd/4.0/ |
dc.subject | Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors |
dc.subject.lcsh | Parallel programming (Computer science) |
dc.subject.lcsh | Big data |
dc.subject.lcsh | Electronic data processing -- Distributed processing |
dc.subject.other | Task-based workflows |
dc.subject.other | Dataflows |
dc.subject.other | Streaming |
dc.subject.other | Convergence HPC-Big Data |
dc.subject.other | Distributed computing |
dc.subject.other | Programming models |
dc.title | A programming model for hybrid workflows: combining task-based workflows and dataflows all-in-one |
dc.type | Article |
dc.subject.lemac | Programació en paral·lel (Informàtica) |
dc.subject.lemac | Macrodades |
dc.subject.lemac | Processament distribuït de dades |
dc.contributor.group | Universitat Politècnica de Catalunya. CAP - Grup de Computació d'Altes Prestacions |
dc.identifier.doi | 10.1016/j.future.2020.07.007 |
dc.description.peerreviewed | Peer Reviewed |
dc.relation.publisherversion | https://doi.org/10.1016/j.future.2020.07.007 |
dc.rights.access | Open Access |
local.identifier.drac | 29009979 |
dc.description.version | Postprint (author's final draft) |
dc.relation.projectid | info:eu-repo/grantAgreement/EC/H2020/730929/EU/Towards an Open, Secure, Decentralized and Coordinated Fog-to-Cloud Management Ecosystem/mF2C |
dc.relation.projectid | info:eu-repo/grantAgreement/MINECO//TIN2015-65316-P/ES/COMPUTACION DE ALTAS PRESTACIONES VII/ |
dc.relation.projectid | info:eu-repo/grantAgreement/AGAUR/V PRI/2014 SGR 1051 |
dc.relation.projectid | info:eu-repo/grantAgreement/MINECO/1PE/BES-2016-076791 |
local.citation.author | Ramón-Cortés, C.; Lordan, F.; Ejarque, J.; Badia, R.M. |
local.citation.publicationName | Future generation computer systems |
local.citation.volume | 113 |
local.citation.startingPage | 281 |
local.citation.endingPage | 297 |
Fitxers d'aquest items
Aquest ítem apareix a les col·leccions següents
-
Articles de revista [317]
-
Articles de revista [1.045]
-
Articles de revista [382]
-
Articles de revista [163]