In this work we propose a solution to some of the problems found in supercomputing environments by combining an extremely efficient, standard, open-source data manager suite with a high-performance data compressor. We do not intend to use such an efficient file format and, later, compress the resulting files or data sets without further ado, as we would be losing in the compression process the file format benefits. Our aim is to compress the little portions of data that conform the data sets stored inside the file (which are named chunks), thus without losing any of the functionalities offered by the mentioned data management suite. HDF5 is our choice for the data storage and management format, and FAPEC is the high-performance data compressor chosen. By integrating FAPEC as an HDF5 filter we will offer a solution that can solve in a smart, clean and efficient way the storage and management problems in supercomputing environments.
In this work we propose a solution to some of the problems found in supercomputing environments by combining an extremely efficient, standard, open-source data manager suite with a high-performance data compressor. We do not intend to use such an efficient file format and, later, compress the resulting files or data sets without further ado, as we would be losing in the compression process the file format benefits. Our aim is to compress the little portions of data that conform the data sets stored inside the file (which are named chunks), thus without losing any of the functionalities offered by the mentioned data management suite. HDF5 is our choice for the data storage and management format, and FAPEC is the high-performance data compressor chosen. By integrating FAPEC as an HDF5 filter we will offer a solution that can solve in a smart, clean and efficient way the storage and management problems in supercomputing environments.
In this work we propose a solution to some of the problems found in supercomputing environments by combining an extremely efficient, standard, open-source data manager suite with a high-performance data compressor. We do not intend to use such an efficient file format and, later, compress the resulting files or data sets without further ado, as we would be losing in the compression process the file format benefits. Our aim is to compress the little portions of data that conform the data sets stored inside the file (which are named chunks), thus without losing any of the functionalities offered by the mentioned data management suite. HDF5 is our choice for the data storage and management format, and FAPEC is the high-performance data compressor chosen. By integrating FAPEC as an HDF5 filter we will offer a solution that can solve in a smart, clean and efficient way the storage and management problems in supercomputing environments.
The Data Compression Group of the Institute for Space Studies of Catalonia (IEEC) has developed FAPEC, the Fully Adaptive Prediction Error Coder. It is a highly optimized adaptive entropy coder, which can be applied as a data compression solution for satellite payloads owing to its very quick and autonomous operation, furthermore with a high resiliency in front of data outliers. FAPEC is also being prepared for on-ground applications as well, for instance within HDF5, a general purpose library and file format for storing scientific data. The project proposed here consists i
