Enabling Python to execute efficiently in heterogeneous distributed infrastructures with PyCOMPSs
Document typeConference lecture
PublisherAssociation for Computing Machinery (ACM)
Rights accessOpen Access
European Commission's projectTANGO - Transparent heterogeneous hardware Architecture deployment for eNergy Gain in Operation (EC-H2020-687584)
Python has been adopted as programming language by a large number of scientific communities. Additionally to the easy programming interface, the large number of libraries and modules that have been made available by a large number of contributors, have taken this language to the top of the list of the most popular programming languages in scientific applications. However, one main drawback of Python is the lack of support for concurrency or parallelism. PyCOMPSs is a proved approach to support task-based parallelism in Python that enables applications to be executed in parallel in distributed computing platforms. This paper presents PyCOMPSs and how it has been tailored to execute tasks in heterogeneous and multi-threaded environments. We present an approach to combine the task-level parallelism provided by PyCOMPSs with the thread-level parallelism provided by MKL. Performance and behavioral results in distributed computing heterogeneous clusters show the benefits and capabilities of PyCOMPSs in both HPC and Big Data infrastructures.
CitationAmela, R. [et al.]. Enabling Python to execute efficiently in heterogeneous distributed infrastructures with PyCOMPSs. A: PyHPC'17 Proceedings of the 7th Workshop on Python for High-Performance and Scientific Computing. "". Association for Computing Machinery (ACM), 2017, p. 1-10.