Testing Kubernetes Clusters - Building Confidence in Your Changes

The talk discusses the challenges of testing Kubernetes clusters and the potential solutions to build confidence in changes made to the clusters and their components.

• Operating Kubernetes clusters involves a large number of components, making it difficult to test the entire setup
• Running the full Kubernetes conformance test suite is slow and doesn't always cover the functionality users care about
• Skyscanner uses Kubernetes' existing test codebase along with Sonobuoy and Spinnaker to build a custom test suite
• The custom test suite allows cluster operators to focus on the functionality developers care about
• The talk provides an overview of potential solutions, including community-supported frameworks like Testemperor and Kubertest
• Kuberhealthy is a slightly different approach that uses an operator to run tests in-cluster and gather results
• Building confidence in changes made to Kubernetes clusters is crucial for reliability and frequent updates

Skyscanner has been running Kubernetes since the 1.6 release and currently has over 35 production clusters spread across four different AWS regions. The clusters run over 475 services, most of which run in multiple clusters and regions for improved resilience. Skyscanner has built up a set of observability components, making it even more challenging to test the entire setup.

Operating Kubernetes clusters involves a large number of components, each of which can impact the functionality of clusters. As cluster operators, building confidence in the changes being made is key to allowing frequent updates without compromising on reliability. Running the full Kubernetes conformance test suite every time is generally too slow, and doesn't always cover the functionality users of clusters care about. In this talk, you will learn how Skyscanner uses Kubernetes' existing test codebase along with Sonobuoy and Spinnaker to build a custom test suite. This allows cluster operators to build confidence in changes made to their clusters and their components, focussing on the functionality developers care about. This case study will enable you to make a start in developing your own tests for Kubernetes clusters, understanding the benefits this can bring, as well as the drawbacks you need to beware of.