Whose Packet Is It Anyway? Life of a Packet Through a Service Mesh

The presentation discusses the use of service mesh and ephemeral containers in Kubernetes for debugging and observability purposes, with a focus on Kafka as an example.

• Service mesh can be used for initial metadata requests in Kafka, but subsequent requests should go directly to specific brokers
• TCP observability is limited in service mesh, so TCP dump and Wireshark can be used for packet captures
• Ephemeral containers can be used to share network namespaces and run debugging tools like TCP dump
• The speaker suggests integrating ephemeral containers into other open source tools like CaseNIc and using them for remote captures on pods

The speaker encountered an issue with a malformed metadata request in Kafka and had to use TCP dump to capture packets and diagnose the problem. They also had to install TCP dump in the container image, which was not included for security reasons.

In this talk, Kevin and Doug will trace a packet through its journey between a meshed client and server. They'll explore how the path of a packet changes after installing a service mesh, the additional hops it introduces, and which networking changes ensure the application's behavior isn't affected.  First they'll observe the networking rule changes that allow for a proxy to intercept traffic. Once we understand what changes about how a packet travels through the kernel, we'll better understand how to observe it in the following steps. Next, in order to observe this packet on its journey they'll take a dive into the Kubernetes networking debugging space. How do you properly use debug containers to observe traffic between other containers? Once you have debugging capabilities, what tools can we use to observe the traffic? Using these tools, attendees will understand what is happening behind the scenes of a service mesh and how a packet travels within it.