Constant-Time Sliding Window Framework with Reduced Memory Footprint and Efficient Bulk Evictions
Visualitza/Obre
Cita com:
hdl:2117/132942
Tipus de documentArticle
Data publicació2019-03-01
EditorIEEE
Condicions d'accésAccés obert
Tots els drets reservats. Aquesta obra està protegida pels drets de propietat intel·lectual i
industrial corresponents. Sense perjudici de les exempcions legals existents, queda prohibida la seva
reproducció, distribució, comunicació pública o transformació sense l'autorització del titular dels drets
ProjecteHi-EST - Holistic Integration of Emerging Supercomputing Technologies (EC-H2020-639595)
COMPUTACION DE ALTAS PRESTACIONES VII (MINECO-TIN2015-65316-P)
COMPUTACION DE ALTAS PRESTACIONES VII (MINECO-TIN2015-65316-P)
Abstract
The fast evolution of data analytics platforms has resulted in an increasing demand for real-time data stream processing. From Internet of Things applications to the monitoring of telemetry generated in large data centers, a common demand for currently emerging scenarios is the need to process vast amounts of data with low latencies, generally performing the analysis process as close to the data source as possible. Stream processing platforms are required to be malleable and absorb spikes generated by fluctuations of data generation rates. Data is usually produced as time series that have to be aggregated using multiple operators, being sliding windows one of the most common abstractions used to process data in real-time. To satisfy the above-mentioned demands, efficient stream processing techniques that aggregate data with minimal computational cost need to be developed. In this paper we present the Monoid Tree Aggregator general sliding window aggregation framework, which seamlessly combines the following features: amortized O(1) time complexity and a worst-case of O(logn) between insertions; it provides both a window aggregation mechanism and a window slide policy that are user programmable; the enforcement of the window sliding policy exhibits amortized O(1) computational cost for single evictions and supports bulk evictions with cost O(logn) ; and it requires a local memory space of O(logn) . The framework can compute aggregations over multiple data dimensions, and has been designed to support decoupling computation and data storage through the use of distributed Key-Value Stores to keep window elements and partial aggregations.
CitacióVillalba, Á.; Berral, J. L.; Carrera, D. Constant-Time Sliding Window Framework with Reduced Memory Footprint and Efficient Bulk Evictions. "IEEE Transactions on Parallel and Distributed Systems", 1 Març 2019, vol. 30, núm. 3, p. 486-500.
ISSN1045-9219
Versió de l'editorhttps://ieeexplore.ieee.org/document/8456588
Col·leccions
Fitxers | Descripció | Mida | Format | Visualitza |
---|---|---|---|---|
Constant-Time Sliding Window Framework2.pdf | 3,008Mb | Visualitza/Obre |