Conference:  KubeCon + CloudNativeCon North America 2021

Authors: Gibbs Cullen

2021-10-14

The debate over stream vs. batch processing has been ongoing for years. While batch processing is optimized for large volumes of data, stream processing allows for real-time analysis. With monitoring workflows aimed at minimizing time to detect incidents, having real-time insights is critical for maintaining reliable cloud-native applications. Monitoring business-critical applications can become difficult at scale. How do you continue processing large volumes of real time data while maintaining valuable insights? There are OSS metrics solutions designed to ingest high volumes of data, but they also need to efficiently aggregate metrics for viewing and analyzing these volumes in real time. This talk will explore how two popular OSS projects, M3 and Thanos, have approached the problem of real time aggregation. The audience will learn how stream and batch processing methodologies have been leveraged by the community to aggregate data in real time, and the tradeoffs of each approach.