Processing and analytics of big data streams with Yahoo!S4

Abstract

Many Internet-based applications generate huge data streams, which are known as Big Data Streams. Such applications comprise IoT-based monitoring systems, data analytics from monitoring online learning workspaces and MOOCs, global flight monitoring systems, etc. Differently from Big Data processing in which the data is available in databases, file systems, etc., before processing, in Big Data Streams the data stream is unbounded and it is to be processed as it becomes available. Besides the challenges of processing huge amount of data, the Big Data Stream processing adds further challenges of coping with scalability and high throughput to enable real time decision taking. While for Big Data processing the MapReduce framework has resulted successful, its batch mode processing shows limitations to process Big Data Streams. Therefore there have been proposed alternative frameworks such as Yahoo!S4, Twitter Storm, etc., to Big Data Stream processing. In this paper we implement and evaluate the Yahoo!S4 for Big Data Stream processing and exemplify through the Big Data Stream from global flight monitoring system.