Conference:  KubeCon + CloudNativeCon Europe 2023

Authors: Kevin Patrick Hannon

2023-04-18

As Kubernetes matures, many users are exploring how and if they should run multi-cluster workloads. Armada is a Sandbox project in the CNCF and its main focus is enabling batch processing across multiple kubernetes clusters. Armada defines APIs for integration and has released a python API. A primary goal of these APIs is to enable integrations with other open source projects. The users at G-Research wanted to use orchestration software to enable the scheduling of multiple tasks on Armada. The open source decided to build an Airflow Operator that allows G-Research users to take advantage of Airflow’s integrations. In this talk, we will briefly introduce Armada and our integration with Airflow. With this integration, Airflow is now able to schedule jobs on multiple Kubernetes.