Kubernetes Defensive Monitoring with Prometheus

Defensive Monitoring in Kubernetes Clusters

• Using metrics from Prometheus to detect anomalies in network traffic and CPU usage
• Manipulating data from Kubernetes metrics to generate topology diagrams of the cluster
• Using service-level metrics to create network topology diagrams
• These techniques can be useful for detecting and investigating security breaches

The speaker gave an example of using the CPU usage metric to detect crypto mining malware installed on a replica in a Kubernetes deployment. By comparing the CPU usage of all replicas, they were able to identify the outlier and investigate further.

A great ecosystem of applications and open source projects has emerged to cover different needs and use cases. However, most of the time we always think about using these applications in the use case that they have been designed for. One example is Prometheus, which is the graduated monitoring project in the CNCF. However, monitoring can become a complementary defensive tool for other projects like Falco. Its access via metrics to other kinds of information that is not available in the kernel calls and the ability to look back in the past, allows Prometheus to cover some blindspots that can be exploited by potential attackers. In this talk, David and Mirco will explore some interesting use cases and practical examples where Prometheus can be used for defensive monitoring, giving some ready to use examples and comparing the pros and cons of this approach with runtime security.