Efficient Model Exploring and Continuous Delivery With Polyaxon + Kubeflow

Conference: KubeCon + CloudNativeCon Europe 2021

Authors: Shotaro Kohama

Summary

Machine learning projects are highly iterative and require smooth transitions from the experimental stage to the productionization phase
Polyaxon is a cloud-native machine learning experiments platform that allows developers to run parallel and scalable hyperparameter tuning job in a declarative way
KubeflowPipelines is a workflow engine for machine learning pipelines
The Machine Learning Platform team at Mercari helps accelerate machine learning projects by Polyaxon and KubeflowPipelines
The team uses these open source tools to automate manual processes and accelerate iterations
The team uses Polyaxon for the model exploring phase and KubeflowPipelines for continuous training
The team also uses Spinnaker for continuous delivery

Melcalli uses machine learning models to design item price lunch and floor plans. They use Polyaxon and KubeflowPipelines to accelerate the iterations and automate manual processes. With these tools, they can manage end-to-end workflows and trigger deployed pipelines when new images are pushed to the registry. This allows them to achieve continuous training and deployment, avoiding early implementation in transition from the modern exploration phase to the continuous training phase. They also built their tools and set up configurations and integrations to be able to write and end the workflow easily.

Abstract

Machine learning projects are highly iterative. A project begins with an experiment phase, a productionization phase, an operation phase, and the next experiment phase. The smooth transition from the experimental stage to the productionization phase is the key to accelerate the iterations. Polyaxon is a cloud-native machine learning experiments platform that allows developers to run parallel and scalable hyperparameter tuning job in a declarative way. KubeflowPipelines is a workflow engine for machine learning pipelines. This talk shows how the Machine Learning Platform team at Mercari helps accelerate machine learning projects by Polyaxon and KubeflowPipelines.

Materials:

Slides

Tags: