Improving CI/CD Process for Cloud Native Python Applications with PyPI Cloud

The presentation discusses the challenges faced in managing a Python Package Index (PyPI) server in a cloud-native environment and explores different options for hosting a PyPI server.

• The speaker discusses the challenges of managing a PyPI server in a cloud-native environment
• The speaker explores different options for hosting a PyPI server, including public PyPI, self-hosted PyPI, and cloud-based PyPI solutions
• The speaker emphasizes the importance of portability, security, resiliency, and speed in a PyPI hosting solution
• The speaker shares an anecdote about the challenges of managing a tangled codebase with embedded machine learning models in multiple services
• The speaker suggests separating the machine learning model portion of the codebase into different repositories and managing them separately as packages in a PyPI server

The speaker shares an anecdote about the challenges of managing a tangled codebase with embedded machine learning models in multiple services, which made it difficult to make changes to the library without triggering the entire pipeline again. The solution was to separate the machine learning model portion of the codebase into different repositories and manage them separately as packages in a PyPI server.

Python remains one of the most popular languages for building data science applications given its simplicity and widespread support for open-source data science projects. While it is easy to integrate such projects into your own Python applications, the CI/CD pipeline of the applications will become heavier and more tightly coupled, even in a modern containerized infrastructure. PyPI Cloud is an open-source private PyPI server which enables users to store and distribute their own Python packages in a cloud-native way. Not only does it provide easy-to-use package management similar to public PyPI, it also helps users build flexible and manageable Python applications by decoupling the CI/CD process of Python packages from application containerization. This talk explores how you can design cloud infrastructure that allows developers to focus on writing interesting Python code without worrying about how to build, deploy, or distribute it. By using a real world example, we will understand strategies to improve each step of the CI/CD pipeline for your Python applications using PyPI cloud.