Enhancing iteration performance on distributed task-based workflows
View/Open
Bercelo et al.pdf (777,3Kb) (Restricted access)
Request copy
Què és aquest botó?
Aquest botó permet demanar una còpia d'un document restringit a l'autor. Es mostra quan:
- Disposem del correu electrònic de l'autor
- El document té una mida inferior a 20 Mb
- Es tracta d'un document d'accés restringit per decisió de l'autor o d'un document d'accés restringit per política de l'editorial
Cita com:
hdl:2117/393645
Document typeArticle
Defense date2023-12
PublisherElsevier
Rights accessRestricted access - publisher's policy
(embargoed until 2024-08-02)
Except where otherwise noted, content on this work
is licensed under a Creative Commons license
:
Attribution-NonCommercial-NoDerivs 4.0 International
ProjecteFlows4HPC - Enabling dynamic and Intelligent workflows in the future EuroHPCecosystem (EC-H2020-955558)
ENABLING DYNAMIC AND INTELLIGENT WORKFLOWS IN THE FUTURE EUROHPCECOSYSTEM (AEI-PCI2021-121957)
ENABLING DYNAMIC AND INTELLIGENT WORKFLOWS IN THE FUTURE EUROHPCECOSYSTEM (AEI-PCI2021-121957)
Abstract
Task-based programming models have proven to be a robust and versatile way to approach development of applications for distributed environments. They provide natural programming patterns with high performance. However, execution on this paradigm can be very sensitive to granularity –i.e., the quantity and execution length of tasks. Granularity is often linked with the block size of the data, and finding the optimal block size has several challenges, as it requires inner knowledge of the computing environment. Our proposal is to supplement the task-based programming model with a new mechanism –our SplIter proposal. At its core, the SplIter provides a transparent way to split a collection into partitions (logical groups of blocks, obtained without any transfers nor data rearrangement), which can then be iterated. Tasks are linked to those partitions, which means that SplIter breaks the dependency between block size and task granularity. The evaluation shows that the SplIter is able to achieve performance improvements of over one order of magnitude when compared to the baseline, and it is either competitive or strictly better (depending on application characteristics) to the competitor alternative. We have chosen different applications covering a wide variety of scenarios; those applications are representatives of a broader set of applications and domains. The changes required in the source code of a task-based application are minimal, preserving the high programmability of the programming model. Two different state-of-the-art task-based frameworks have been evaluated for all the applications: COMPSs and Dask, showing that the SplIter can be effectively used within different frameworks.
CitationBarcelo, A.; Queralt, A.; Cortes, A. Enhancing iteration performance on distributed task-based workflows. "Future generation computer systems", Desembre 2023, vol. 149, p. 359-375.
ISSN0167-739X
Publisher versionhttps://www.sciencedirect.com/science/article/pii/S0167739X23002911
Other identifiershttps://arxiv.org/abs/2308.03480
Files | Description | Size | Format | View |
---|---|---|---|---|
Bercelo et al.pdf![]() | 777,3Kb | Restricted access |