Conference:  OpenAI + Data Forum 2022

Authors: Karin Wolok, Rong Rong

2022-06-21

The presentation discusses how to build a dashboard using Kafka and Pinot to analyze Wikipedia data in real-time.

• Building a dashboard using Kafka and Pinot to analyze Wikipedia data in real-time
• Using Streamlit to create metrics and charts for the dashboard
• Demonstrating how to drill down and analyze data using the dashboard
• Auto-refreshing the dashboard to display real-time data

Conference:  KubeCon + CloudNativeCon North America 2021

Authors: Gibbs Cullen

2021-10-14

The debate over stream vs. batch processing has been ongoing for years. While batch processing is optimized for large volumes of data, stream processing allows for real-time analysis. With monitoring workflows aimed at minimizing time to detect incidents, having real-time insights is critical for maintaining reliable cloud-native applications. Monitoring business-critical applications can become difficult at scale. How do you continue processing large volumes of real time data while maintaining valuable insights? There are OSS metrics solutions designed to ingest high volumes of data, but they also need to efficiently aggregate metrics for viewing and analyzing these volumes in real time. This talk will explore how two popular OSS projects, M3 and Thanos, have approached the problem of real time aggregation. The audience will learn how stream and batch processing methodologies have been leveraged by the community to aggregate data in real time, and the tradeoffs of each approach.