Implementing an Auditable Access Control Strategy Using Cluster Certificate Authority Rotation

The presentation discusses the process of introducing a new cluster CA in a Kubernetes environment and updating server-side components and certificates to avoid downtime and maintain security.

• Introducing a new cluster CA is necessary for security and compliance purposes
• The process involves updating server-side components and certificates in a well-defined multi-step process
• Cross-signed CA certificates are used to maintain existing MTLS connections
• The new CA is rolled out across all server-side components
• The client and server certificates are updated to be issued from the new CA
• The cross-signed new CA certificate by the old CA is included in the chain to validate existing MTLS connections

The presentation uses the example of an audit where the administrator group has left the organization and the auditors ask for evidence of credential and secret rotation and reapproval of existing administrators. This leads to the start of the CA rotation process.

Changes in staff and credential exposures require organizations to have an enforcable strategy to revoke and renew access to Kubernetes clusters. Cluster certificate authorities need to be rotated in addition to the downstream certificates these cluster CAs sign to implement an access renewal and revocation strategy. Certificates issued by the cluster CA including node kubelet client certificates, node kubelet server certificates, and cluster administrator certificates need to be able to be rotated in a zero downtime fashion in order to maintain availability throughout this revocation process. This talk outlines a strategy that organizations can utilize to rotate cluster CAs with zero downtime using CA cross signing. We will visually walkthrough the workflow and how certificates for individual critical cluster components change throughout the rotation process. We will touch on how cross signing enables this process to occur without any downtime to existing components running within the cluster. We will then touch on how new access can be granted to the cluster once the rotation process is complete.