KubeCon + CloudNativeCon Europe 2023

Kubernetes is no longer just running service oriented workloads. Batch and data processing workloads are everywhere. Apache Spark is a good example. Scheduling requirements differ for these different types of workloads. The default scheduler and workload resources, like a job or cronjob, are not always a good fit. Jobs change each run based on the data being processed. Some jobs need a set of pods, or gang, to start processing, but could use more or less. How do we get that flexibility? Pods are often created on the fly by a management pod. Jobs can fail due to pod rejection as part of quota enforcements. Why not queue the pods instead until quota is available? What happens if the management pod get preempted? Using the scheduling framework Apache YuniKorn has extended the scheduler to add batch focussed functionality. All added functionality is opt-in and can be used on top of the existing default functionality. Job aware preemption, gang scheduling and queueing with quotas enforced while scheduling are some of the features we will cover.

Dates

Author

Conferences

Tags

Scheduling Framework: Batch Extensions with Apache YuniKorn

tldr - powered by Generative AI