Debugging at Scale in Production - Deep into Your Containers with Kubectl Debug, KoolKits and Continuous Observability

The presentation discusses the challenges of debugging in a Kubernetes environment and introduces Cube City or Debug and Cogito as solutions.

• Debugging in a Kubernetes environment is challenging due to multiple layers of abstraction and the bare bone container problem.
• Cube City or Debug and Cogito are solutions to these challenges.
• Cube City or Debug allows for inspection of a pod even if it has crashed or is a bare bone image.
• Cogito is an open source project that includes a set of opinionated curated platform-specific tools for debugging with Cube City or Debug.
• Anecdote: The presentation provides a demo of using Cube City or Debug to increase logging levels and connect to an ephemeral container with the Busybox image.
• Tags: Kubernetes, debugging, Cube City or Debug, Cogito, ephemeral container, bare bone container problem.

Brian Kernigham said: “Debugging is twice as hard as writing the code in the first place.” In fact, debugging in a modern production environment is even harder - orchestrators spinning containers up and down and weird networking wizardry that keeps everything glued together, make understanding systems that much more difficult than it used to be. And, while k8s is well understood by DevOps people by now, it remains a nut that developers are still trying to crack. Where do you start when there’s a production problem? How do you get the tools you’re used to in the remote container? How do you understand what is running where and what is its current state? In this talk, we will review debugging a production application deployed to a Kubernetes cluster, and review kubectl debug - a new feature from the Kubernetes sig-cli team. In addition, we’ll review the open source KoolKits project that offers a set of (opinionated) tools for kubectl debug. KoolKits builds on top of kubectl debug by adding everything you need right into the image. When logging into a container, we’re often hit with the scarcity of tools at our disposal. No vim (for better or worse), no DB clients, no htop, no debuggers, etc… KoolKits solves those problems in an elegant way...