Building Container Images In Kubernetes: It’s Been a Journey!

The presentation discusses the challenges faced by Datadog in migrating their container image builds to Kubernetes without additional privileges, and how they overcame these challenges using buildkit in rootless mode.

• Datadog migrated their container image builds to Kubernetes to overcome limitations with Docker machine and to support arm binaries
• Building container images in rootless mode using buildkit worked for over 90% of their images, but they encountered complex issues for the remaining 10%
• One issue involved SC Linux attributes on files, which could not be modified in a user namespace used by buildkit D
• Another issue involved a timeout error caused by a port being bound by the abdomen
• Datadog used netstat to debug the issues and worked with the community to address them

The presentation shared an anecdote about how they traced the build kit D daemon and found that an operation not permitted message was caused by an SC Linux attribute on a file. They downloaded the layers of the image and extracted the turbo to confirm that there were SC Linux labels on the files. They opened an issue upstream and recommended removing the SC Linux attributes or using an image without any SC Linux label.

Almost all of Datadog now runs on Kubernetes, but for a long time we needed dedicated nodes running Docker to build container images. We have recently migrated container image builds to Kubernetes and it's been an interesting journey!The main challenge to build container images inside Kubernetes is to achieve it without additional privileges. We will explain why we chose buildkit in rootless mode, the architecture we ended up using, as well as the challenges we faced.Building container images in rootless mode worked flawlessly for over 90% of our images, but for the remaining 10% we encountered complex and interesting issues. We will dive into these problems and explain in detail how rootless builds work and why they sometimes behave differently. We will also explain how we addressed these issues together with the community.