KubeCon + CloudNativeCon North America 2022

Three years ago ING Wholesale Banking Advanced Analytics team set up an ambitious goal to gather in one place a curated portfolio of internal data sources together with a large scale compute platform. At its core the idea of allowing internal projects to get access to a rich toolset of open source and industry standards frameworks and preprocessed data to validate business ideas in the secure exploration environment. Extensive growth with over 300 internal projects so far and more than 2000 internal users proofs advanced analytics i.e. ML, AI, NLP capabilities should become easily consumable not only by specialized, dedicated teams, but make them close to subject matter experts. In this session we would like to shed more light on how a specialized cloud native Kubernetes scheduler (Volcano) enables us to deliver multi-tenant large scale processing capabilities. The optimal resource usage with stability of core services are key for our cloud native platform. To enable dynamic allocation and hdrf (hierarchical dominant resource fairness) we have created an extension to Apache Spark binaries. This allows users to use Volcano with Spark interactive mode in a Jupyter notebook. Additionally we have created interfaces to visualize all the scheduling metrics like the yarn ui.

Dates

Author

Conferences

Tags

Efficient Scheduling Of High Performance Batch Computing For Analytics Workloads With Volcano

tldr - powered by Generative AI