How We Migrated Over 1000 Services to Backstage Using GitOps and Survived to Talk About It!

Conference: KubeCon + CloudNativeCon Europe 2023

2023-04-20

Authors: Shahar Shmaram, Ran Mansoor

Summary

The presentation discusses the challenges faced by a company during hyper growth and how they implemented a solution using GitHub's methodology and Backstage to manage their resources and visualize them in one place.

The company faced challenges during hyper growth such as lack of alignment, manually managed resources, unknown resource dependencies and ownership, exploding budget, and lack of technical documentation.
They implemented a solution using GitHub's methodology which emphasizes declarative infrastructure as code, versioning, immutability, automatic deployment pipelines, and continuous reconciliation.
They also used Backstage, an open platform for building developer portals, to manage their resources in one location, write documentation easily, search for information, use automated software templates, and create self-contained plugins.
The solution was auditable, declarative, had a single source of truth, was community-driven, self-serve, and provided visibility.
An anecdote was given about how the GitHub solution detected a drift in a policy and automatically brought it back to its desired state.
Tags: AI, Cybersecurity, DevOps, GitHub, Backstage, hyper growth, resource management, visualization, automation, documentation, self-serve, community-driven.

During the presentation, an anecdote was given about how the GitHub solution detected a drift in a policy and automatically brought it back to its desired state. The presenter explained that they had a policy in place that required all resources to be tagged with a specific key, but someone had removed the tag from a resource. The GitHub solution detected the drift and automatically applied the desired state, which included the tag. The presenter emphasized the importance of having an auditable solution that can detect and correct drifts automatically to avoid manual errors and ensure compliance.

Abstract

We decided to go ALL IN with GitOps and Terraform so all developers in AppsFlyer now manage their cloud resources in their git repository, later we realized it was time to have a single source of truth catalog for the entire company’s assets that can be self-extended by any of AppsFlyer's developers, we decided to choose Backstage as our service catalog platform. But how do we integrate them both together? AppsFlyer has over 1000 services and processes HUGE amounts of data daily (over 200 Billion events per day). Our customers depend on us to help them make good choices, so while we knew that this needed to be done, we also knew that the migration process was not going to be an easy task. As if things weren't complicated enough, we needed to integrate GitOps and Terraform into that process. In this talk, we will demonstrate how we managed to enable GitOps in the migration process within Backstage. We’ll discuss different approaches we took, the challenges we faced, and most importantly, our unique approach to solving them.

Materials:

Tags: