Beyond printf and tcpdump: Debugging Kubernetes Networking with eBPF

Debugging Kubernetes networking issues with eBPF

• Debugging Kubernetes networking can be difficult due to the complexity of Linux kernel networking
• Traditional tools like tcpdump and logging-based methods are not sufficient for debugging
• eBPF can be used to efficiently troubleshoot K8s networking issues
• Packet inspection across layer 2, layer 3 and policy routing, socket, and so on, regardless of the CNI
• Real-life examples of K8s networking problems and how they were debugged with eBPF

The speaker shared a story about a multi-homing setup in a Kubernetes cluster where a pod was assigned IP addresses from two different IP subnets. The pod was trying to reach the cube API server part, but the reply never reached the source due to incorrect routing. Using eBPF and a set of filters, the speaker was able to trace all the kernel functions that the traffic was going through and identify the issue with the source IP address.

Debugging Kubernetes networking issues can easily turn into a nightmare. Packets from one pod to another can traverse a couple of dozen Linux kernel functions. Each function execution outcome might depend on an OS state which is not exposed to a user and is not visible with traditional Linux networking tooling. And those tools can’t tell you how network packets relate to K8s pods and services. Luckily, with the help of eBPF we no longer need to treat Linux kernel networking in the context of K8s as a big blackbox. In this talk DevOps who operate K8s clusters will learn: - Packet inspection across layer 2, layer 3 and policy routing, socket, and so on, regardless of the CNI. - How eBPF can be used to efficiently troubleshoot K8s networking issues. - Real-life examples of K8s networking problems and how they were debugged with eBPF. No previous knowledge of kernel internals or deep understanding of networking is required to attend the talk.