Breakpoints in Your Pod: Interactively Debugging Kubernetes Applications

The presentation discusses the challenges of debugging Kubernetes applications, particularly Cluster API, and how to solve them using ephemeral containers and open-source tools.

• Debugging Kubernetes applications is challenging due to the lack of a debugger in the pod and the separation of the target executable and process from the debugger client
• Ephemeral containers can be used to run the debugger in the same process namespace as the target and to avoid including unnecessary utilities in the container image
• The presenter demonstrates how to use an IDE to simultaneously debug multiple controllers in Cluster API using breakpoints
• The use of widely-available, open-source tools such as the Delve debugger and Alpine Linux image makes interactive debugging of Kubernetes applications more accessible

The presenter shares a demo of debugging multiple controllers in Cluster API using breakpoints set in three different pods. The breakpoints help track the creation of a new machine in the cluster, and the demo showcases the use of ephemeral containers and an IDE to achieve this.

Interactive debugging has been around for decades. Being able to set breakpoints, watch expressions, and “stepping” through code not only helps investigate bugs, it helps us learn how complex applications work. Debugging an application in Kubernetes is like debugging a remote process, but with more challenges. Container images frequently have executables without debug symbols, and no debugger server executable.The typical Pod does not run a debugger server, and does not expose a port on which a debugger client could connect. The target executable is in a separate mount namespace, and the target process in a separate process namespace. This session will demonstrate how to solve these challenges by using ephemeral containers, and widely-available, open-source tools. Along with these tools, we will use an IDE to simultaneously, interactively debug multiple, cooperating controllers from the Cluster API project.