100Gbit/S Clusters With Cilium: Building Tomorrow’s Networking Data Plane

The talk is about the possibility of IPv6-only clusters addressing scale and performance requirements in data center networks.

• Psyllium experiment started in 2016 with IPv6-only container networking using EBPF and XDP
• IPv6 adoption has progressed in Kubernetes and hyperscale environments
• IPv6 offers more IPAM flexibility and larger cluster scale
• IPv6-only clusters unlock new Linux kernel innovations in networking and EBPF for data intensive workloads
• Cilium's networking data plane enables a low-latency architecture suitable for BIG TCP-based workloads requiring IPv6 for 100Gbit/s transfers and beyond for a single socket
• Cilium developed a new veth driver replacement for the kernel to achieve host networking performance characteristics for Pods
• With the resulting EBPF forwarding architecture, most unneeded parts of the stack are bypassed, drastically improving networking

In 2016, Psyllium experiment started with IPv6-only container networking using EBPF and XDP. However, the state of IPv6 adoption in Kubernetes and Docker was not quite there yet, so they had to implement IPv4 support upon popular demand. Fast forward to 2020, IPv6 adoption has progressed in Kubernetes and hyperscale environments, and IPv6 offers more IPAM flexibility and larger cluster scale. Cilium's networking data plane enables a low-latency architecture suitable for BIG TCP-based workloads requiring IPv6 for 100Gbit/s transfers and beyond for a single socket. They also developed a new veth driver replacement for the kernel to achieve host networking performance characteristics for Pods. With the resulting EBPF forwarding architecture, most unneeded parts of the stack are bypassed, drastically improving networking.

The vast majority of Kubernetes users today are running IPv4-only clusters or transitioning to dual stack deployments as an interim step on the journey towards IPv6-only clusters. The latter are not only an enabler for more IPAM flexibility and larger cluster scale, but also unlock new Linux kernel innovations in networking and eBPF to cater for data intensive workloads. In this talk, we examine what's possible once we arrive at IPv6-only clusters. We present recent advancements in Cilium's networking data plane enabling a low-latency architecture suitable for "BIG TCP"-based workloads requiring IPv6 for 100Gbit/s transfers and beyond for a single socket. In addition, we deep dive our path into achieving host networking performance characteristics for Pods through a new veth driver replacement we specifically developed for the kernel in context of Cilium. With the resulting eBPF forwarding architecture, most unneeded parts of the stack are bypassed, drastically improving networking.