K8s in a Submarine: Optimizing Delivery for Some of the Hardest-to-reach Systems on Earth

The presentation discusses the challenges faced by the speaker's team while delivering applications in K8s for the US Navy in air-gapped environments. The focus is on the lifecycle for bringing software declaratively into K8s and the need for predictably packaging all dependencies.

• Air-gapped environments pose unique challenges for delivering applications in K8s
• Declarative state is important for reproducing environments over and over
• Redundancy is necessary to accommodate bugs and failures
• Tools like Helm, Argo, Flux, and kubectl only address half of the problem by managing manifests and leaving the images they depend on unmanaged
• Zarf, an open-source tool, was built to extract images from manifests, charts, kustomizations, and operators and package all dependencies predictably
• Creating a registry temporarily using a rust binary and config maps was a solution to the chicken and egg problem
• The speaker's team had experience deploying K8s in various air-gapped environments, including on fighter jets and submarines

The speaker had experience deploying K8s on top secret networks, classified secret networks, fighter jets, and submarines. The most extreme case was the submarine, which may be submerged and have no connectivity beyond very low frequency communication. The lack of internet connection, stack overflow, stack exchange, and GitHub posed unique challenges for delivering applications in K8s.

In this talk, Jeff will walk through the problems faced by his team while delivering applications in K8s for the US Navy. There are many established patterns for creating clusters in air-gapped environments, so this talk will focus less on creating a cluster and more on the lifecycle for bringing software declaratively into K8s. Jeff will discuss operating an in-cluster registry, issues with HA, routing, DNS, TLS, and the Node/CRI relationship with in-cluster resources. He will also explain various methods they explored for "pushing" images into a cluster with no external registry and how various K8s distros deal with this problem. Additionally, Jeff will discuss how gitops-based deployments pose unique challenges and why they chose tools such as Gitea. Jeff will also discuss how tools such as Helm, Argo, Flux, and kubectl only address half of the problem by managing manifests and leaving the images they depend on unmanaged. He will explore ways to extract images from manifests, charts, kustomizations, and even operators in some cases. Finally, Jeff will discuss the need for predictably packaging all of these dependencies and some of the tools they evaluated for doing so before ultimately deciding to build the open-source tool, Zarf, in partnership with the US Navy.