KubeCon + CloudNativeCon Europe 2023

Intuit runs ~2500 services on Kubernetes, and being one of the top SAAS companies; operational excellence is a top priority. While considerable effort and cost were spent on all three pillars of observability (metrics, log, events), there was a gap in identifying customer impact and causal service to improve MTTD/R for our services. The challenge is that we need to analyze massive amounts of data generated by these data sources in real-time at scale, which is a beast of a problem. Our talk focuses on using Numaproj (Intuit’s open project) and other CNCF open-source technologies to address this problem. Numaproj includes Numaflow, a stream processing platform, and Numalogic, a collection of ML models. We will show how we collect, process, and analyze data per minute in real-time and how using Numaproj, we compute normalized anomaly scores for every data point. These anomaly scores helped us weed the noises out from the data, provide a high signal-to-noise ratio, and directly create incidents. Today we detect incidents based on the scores (~98% confidence) generated by our AIOps platform. With this solution, Intuit has established a cookie-cutter to reduce its MTTD from over 30 minutes to less than 3 minutes.

Dates

Author

Conferences

Tags

Intuit’s Customer Centric Observability Journey Using AIOps

tldr - powered by Generative AI