AbacusFS integrated storage and computational platform

Abstract

Today's applications, especially those in the scientific community, deal with an ever growing amount of data. Among the problems that arise from this explosion of data are how to organize the data so that the information about how the data was produced is not lost, how to ensure repeatability of calculations, how to automate calculations, and how to save computational and storage resources. The storage is currently a passive component that neither imposes any rules on data organization nor helps with associating a calculation with its result. We thus introduce AbacusFS – an integrated storage and computational platform which interacts with the file system and connects the calculations and the storage. Users and applications see it as a normal file system; however, the semantics are changed for files that are results of calculations done with the platform. When such a file is open for reading, AbacusFS first regenerates it if the input files used in the calculation have been modified. Alternatively, result files can be virtual, i.e. not stored anywhere but rather regenerated on every read access.