How to Leverage Volcano to Improve the Resource Utilization of AI Pharmaceuticals, Autonomous Driving, and Smart Buildings

Conference: KubeCon + CloudNativeCon Europe 2023

2023-04-21

Authors: William Wang

Summary

Volcano is a cloud-native batch system that provides a unified job scheduling and management solution for Kubernetes clusters. It is designed to be scalable, flexible, and extensible, and it supports a wide range of workloads, including machine learning, data processing, and scientific computing.

Volcano is a cloud-native batch system that provides a unified job scheduling and management solution for Kubernetes clusters.
It is designed to be scalable, flexible, and extensible, and it supports a wide range of workloads, including machine learning, data processing, and scientific computing.
Volcano has several features that make it a powerful tool for managing batch workloads, including job scheduling, resource management, and job dependencies.
Volcano is used by a diverse group of users, including those in the AI and data areas, and it has a large and active community of contributors.
Volcano integrates with a variety of other tools and platforms, including Spark, Argo, and Airflow.
Volcano provides documentation and support for a wide range of training operators, including TensorFlow, MXNet, and MPI.

One user of Volcano is a drug discovery workload that requires massive computing power. Volcano worked with them to develop a global scheduling based on Volcano and provided cluster load balancing impacting and cluster affinity. Users also use the Volcano job to run their testing flow and pathology and MPI workload uniformly.

Abstract

Volcano is a cloud-native open source system for running high-performance workloads on Kubernetes. With community support, the project is becoming the platform of choice for many users and continues to grow as an incubating project. In the year 2022, Volcano helps users to support more scenarios like AI pharmaceuticals, autonomous driving, and smart buildings fileds. In this talk, William will review Volcano motivation, architecture; update you on Volcano's progress towards 1.8, the latest deployments, such as jobflow and GPU sharing. then go through latest updates on new features, user adoptions, and new subprojects. After that William will introduce where the project is heading to, updated project roadmap and how new contributors to get involved. There will be an open Q&A for attendees to ask questions.

Materials:

Tags: