The CRDs that Broke the Camel's Back

The talk discusses the performance issues related to the API server when installing thousands of CRDs and how to troubleshoot them using profiling tools. It also provides insights into the mechanics of CRDs and tips for getting changes into upstream.

• Custom resources are used to extend the K8s API server with a declarative API
• Initial attempts to install thousands of CRDs revealed severe performance issues related to the API server
• Profiling tools can be used to troubleshoot API server performance issues
• Real world data can help pinpoint the root causes of scaling issues
• Insights into the mechanics of CRDs are provided
• Tips for getting changes into upstream and moving the ecosystem forward are shared

The speaker uses the idiom 'the straw that broke the camel's back' to explain how one thing can trigger a reaction and cause a situation to become unbearable. This idiom can be used in any field, such as politics or sports, to describe a situation where one thing becomes the tipping point.

Custom resources are the preferred way to extend the K8s API server with a declarative API. They enable us to implement our very own control planes on top of K8s. K8s has performance guidelines and thoroughly investigated scalability thresholds but no guidelines for CRDs are available yet. Our initial attempts to install 1000s of CRDs revealed severe performance issues related to the API server, such as service disruptions and client-side throttling. And this further led to investigations to reveal the root causes of those issues. This talk aims to discuss how one can troubleshoot API server performance issues using profiling tools and to present some real world data that allowed us to pinpoint the root causes of the scaling issues that we initially hit. As the troubleshooting process is explained, the talk will also deliver some insights into the mechanics of CRDs. We would also like to share some tips in successfully getting changes into upstream and moving the ecosystem forward.Click here to view captioning/translation in the MeetingPlay platform!