Conference:  KubeCon + CloudNativeCon Europe 2023

Authors: Leila Vayghan

2023-04-19

This talk is a story of how Shopify runs a highly available and scalable stateful application on Kubernetes which is accessed securely over the internet. The application discussed is Elasticsearch which stores petabytes of data over the globe. Search is a fundamental component of an ecommerce platform and high availability is an important requirement for it. While Kubernetes has proven to be the perfect platform for deploying stateless applications, running stateful applications on this platform in a highly available and scalable manner can be complicated. This talk will discuss these challenges and will share the steps towards solving them. For example, Leila will explain the obstacles of implementing storage autoscaling and how using the existing Kubernetes features allowed seamless expansion of persistent disks that store critical search data. She will also explain how her team implemented a feature that allowed shrinking persistent disks without any data loss and saved costs by releasing unused storage. Leila will also explain how Envoy is used to allow clients to connect to Elasticsearch through Kubernetes' ingress. This talk will give insight into the challenges and rewards of running highly available and scalable stateful applications on Kubernetes.

Conference:  CloudOpen 2022

Authors: Laysa Uchoa

2022-06-22

OpenSearch is a community-driven, open-source fork of Elasticsearch and Kibana. Developers who are curious about how search in OpenSearch works will be welcome in this session. Finding a four-pawed friend to adopt can be challenging. If you live in a small flat, you may have some requirements regarding the size of pet you can accommodate. But can OpenSearch assist you in this search? We will try! In this talk, we will learn how to write and run search queries on our OpenSearch cluster with the purpose to find your future friend. We will cover the most common queries from term-level, full-text queries and boolean queries. We will be playing with an interesting pet dataset and the Python OpenSearch client.

Conference:  KubeCon + CloudNativeCon Europe 2022

Authors: Ciprian Hacman, Radu Gheorghe

2022-05-19

Best practices for scaling Elasticsearch clusters

• Use metrics from inside Elasticsearch for accuracy
• Scale in larger increments to reduce noise
• Force index rotation to evenly spread load across nodes
• Judge cluster size based on disk usage and search latency
• Use local SSDs for better I/O latency
• Consider hot-warm-cold architecture for data management