tldr - powered by Generative AI

• Databricks is building a serverless platform for performance-sensitive workloads such as Data Lakehouse on Kubernetes clusters
• They need hard multi-tenant container isolation since each cluster runs code on behalf of multiple customers
• They chose Kata Containers, an open-source container runtime that provides strong isolation by running containers in micro-VMs
• They built a hard compute and network isolation layer among untrusted workloads in Kubernetes clusters leveraging Kata Containers, network policy, and network security group
• They share their first-hand experience on how they integrate Kata Containers with Kubernetes in production, highlighting the challenges they faced, difficult trade-offs among security, performance, and cost, and how to work around the heterogeneity across different public cloud providers