Analyzing European Deep-Learning libraries with Industry Standard Benchmark

Abstract

For the past decade, machine learning (ML) has revolutionized numerous domains in our daily life. Nowadays, deep learning (DL) algorithms are the central focus of modern ML systems. As a result, we are witnessing an impressive surge of DL libraries and customized hardware designs that can efficiently support this computing-intensive techniques. The ever increasing development of new libraries and hardware designs rises a new challenge: how to create a fair comparison between the various implementations of DL libraries exploiting different hardware designs. Being able to compare these solutions is a key element to drive the design of new features and better exploit different hardware designs. To improve DL libraries, we need to adopt a common benchmark for the evaluation of the many DL frameworks in terms of accuracy and performance. A standardized benchmark would produce a fair and comprehensive evaluation of all the available methods, paving the way for better hardware-specific designs. However, benchmarking ML methods presents a number of unique challenges. First, some optimization techniques improve training accuracy at the cost of increasing "time to solution". Also, training is an stochastic process and "time to solution" depicts a high variability. Last, there are multiple software and hardware systems and it is not straightforward how to fairly evaluate them under the same parameters. MLPerf training benchmark defines guidelines to mitigate these challenges. MLPerf's mission is to build a fair and useful benchmarks for measuring performance of ML hardware, software, and services. The benchmark measures how fast systems can train models to a target quality. The first release (v0.5) contains seven different DL suites across four domains: (1) vision, (2) language, (3) commerce, and (4) research. Each of these suites is defined by a dataset, a quality target, and a reference implementation model. In this work, we present an image classification evaluation based on MLPerf benchmark for three DL libraries: Keras-TF, PyTorch, and the European Distributed Deep Learning Library (EDDLL). Keras and PyTorch have become very popular and widely-used libraries in the DL field. Meanwhile EDDLL is a novel library being developed as a part of the European project DeepHealth. Our goal is to perform a unified performance evaluation and characterization of these DL frameworks using the High Performance Computing (HPC) infrastructures of the BSC. The present work aims to help improving the EDDL library. We dive into the core implementation of the library to identify potential bottlenecks and explore optimization opportunities. This way, this work establishes the foundations for the EDDL CPU-optimized version, adapted to the HPC hardware infrastructure without compromising the accuracy results of the benchmark. As a result of this work, the EDDL CPU-optimized version achieves a 2x speedup over the baseline version, executed on Intel Xeon Platinum processo