Distributing PromQL for Fast and Efficient Kubernetes Fleet Monitoring

The presentation discusses the implementation of vertical sharding in Thanos for efficient query processing and scaling.

• Vertical sharding can be useful for Prometheus as well as Thanos.
• Sharding is implemented end-to-end from query execution to data retrieval from the store.
• Sharding queries across a fleet of Thanos queries allows for horizontal scaling.
• There is a proposal in upstream Thanos for grouping and aggregation queries.
• Contributing to Thanos is encouraged and the community is friendly.

The presenters explain that Thanos queries can be horizontally scaled by sharding queries across a fleet of Thanos queries. This allows for a more efficient use of resources and easier scaling up and down. They also mention that there is ongoing work to add sharding at the tsdb level in Prometheus. The presenters encourage the audience to contribute to Thanos and engage with the community.

Both Thanos and Cortex have enabled the cloud native ecosystem to scale Prometheus storage with the use of blocks of data persisted across many clusters into single object storage. Whilst this unlocks cheap long term retention of metrics, it presents a significant challenge of being able to efficiently read and process large volumes of data. This talk outlines the Thanos community's efforts to improve read path performance through query pushdown and query sharding and how it compares with existing Cortex approaches. Thanos deployment's are composed of stores; components that expose a consistent Prometheus compliant read API for retrieving timeseries, and queriers; components that combine raw timeseries and evaluate PromQL expressions against them. Query pushdown gives the opportunity to pre-evaluate these expressions closer to the data, while query sharding breaks down a query into distinct, disassociated datesets that can be computed concurrently thanks to Kubernetes.Click here to view captioning/translation in the MeetingPlay platform!