Availability and Storage Autoscaling of Stateful Workloads on Kubernetes

This talk is a story of how Shopify runs a highly available and scalable stateful application on Kubernetes which is accessed securely over the internet. The application discussed is Elasticsearch which stores petabytes of data over the globe. Search is a fundamental component of an ecommerce platform and high availability is an important requirement for it. While Kubernetes has proven to be the perfect platform for deploying stateless applications, running stateful applications on this platform in a highly available and scalable manner can be complicated. This talk will discuss these challenges and will share the steps towards solving them. For example, Leila will explain the obstacles of implementing storage autoscaling and how using the existing Kubernetes features allowed seamless expansion of persistent disks that store critical search data. She will also explain how her team implemented a feature that allowed shrinking persistent disks without any data loss and saved costs by releasing unused storage. Leila will also explain how Envoy is used to allow clients to connect to Elasticsearch through Kubernetes' ingress. This talk will give insight into the challenges and rewards of running highly available and scalable stateful applications on Kubernetes.