Discontiguous CIDRs for Dynamic Cluster Scaling

Enhancing Kubernetes Node IPAM to support multiple cluster cider ranges

• Two modes of operation for Node IPAM: node allocator and cloud allocator
• Proposal to enhance Node IPAM to support multiple cluster cider ranges
• Introduction of a new Kubernetes resource called the cluster cider config
• New allocator to watch the cluster cider config object and perform node IP allocations based on that information
• Ability to specify discontiguous ranges for pods and accommodate different node sizes

The proposal aims to solve the problem of fragmented IP space by allowing users to specify discontiguous pod ranges at cluster startup time. This addresses the challenge of accommodating different node sizes and efficiently using IPs across nodes. The dependency on a single contiguous cider block is being removed from Kubernetes components such as the Node IPAM controller and the Cube Proxy. For instance, the Cube Proxy uses the cluster cider to make routing decisions and prevent traffic from unnecessarily egressing the cluster. The proposal offers a more useful operation right out of the box, where users don't need to configure their own IPAM controller to expand their cluster slider size.

Users want more out of their existing clusters. As demand increases, they would like to increase the size of their clusters by adding more nodes and pods (scaling horizontally). One of the key limitations in doing this is immutability of IP addresses. Providing each additional node and pod an IP, quickly exhausts users’ initial IP allocations. They must resort to optimizing IP addresses in their existing clusters, or pre-emptively over-allocating IP addresses during cluster creation time. With Discontiguous Pod and Node CIDRs, IP addresses become a mutable resource that can be added post-cluster creation thereby helping in the growing the cluster dynamically. Adding support for disjoint CIDRs also allows fitting clusters into fragmented IP spaces, a problem commonly faced by enterprise customers. This talk will describe how GKE solves the problem for customers and the lessons learnt along the way. This talk also proposes ideas on how to solve this natively in Kubernetes.