A Large-Scale, End-to-End Synchronous, Distributed Data Stream Processing Architecture for Time-Sensitive Applications

Abstract

Recent research works when benchmarking Data Stream Processing and IoT platforms, report latencies on the order of tens of seconds and thus they are not suited for time-critical applications. Such platforms mainly use loosely coupled components with asynchronous communication. In this work, we present an alternative novel approach. We propose a novel End-to-End Synchronous, Distributed Architecture for Large-Scale, High-Bandwidth, Ultra-Low Latency Data Stream Processing which achieves latencies on the order of milliseconds. Through our Big Data Stream analysis experiments performed over the ARIS High Performance Computing facility of the National Infrastructures for Research and Technology S.A. (GRNET S.A.) - an EGI Council Participant, we have demonstrated the suitability of our architecture for time-sensitive applications.

Panayiotis A Michael (National Technical University of Athens & University of California Los Angeles, Greece); Panayiotis Tsanakas (National Technical University of Athens, Greece) and D. Stott Parker (University of California, Los Angeles, USA) has presented the latest PolicyCloud paper, in the 2022 edition of the IEEE/ACM 26th International Symposium on Distributed Simulation and Real-Time Applications (DS-RT) held in Alès, France, during the Real-time session at 15:30 CEST, tackling "A Large-Scale, End-to-End Synchronous, Distributed Data Stream Processing Architecture for Time-Sensitive Applications".