How Spotify Leverages Flyte To Coordinate Financial Analytics Company-Wide

The presentation discusses the differences between DevOps and ML Ops, and how Flight can help with ML Ops workflows.

• ML Ops has different requirements than DevOps, such as dealing with large amounts of data and longer processing times
• Flight is a platform that can help with ML Ops workflows, providing features such as composition, caching, and validation
• Flight can handle individual containers for task execution environments, but not yet streaming data
• Flight has been well-received by data scientists for its ease of use and out-of-the-box parallelism

The data science team at Spotify found Flight to be a big win for their workflows, particularly for its ability to handle composition and parallelism. One team even built a project with common tasks that everyone could use. Flight also emphasizes early error detection through type checks and validation. Oktoberfest is an event where users can contribute to Flight and win swag.

Kubernetes’ popularity stems from its declarative yaml based API and focus on versioning and other best practices of developing micro-services. Therein also lies its weakness, where users that are not conversant with infrastructure are inadvertently discouraged from leveraging the power of Kubernetes. Using an example application at Spotify that powers their financial platform, the talk examines: * The specific operational needs of ML and Data Engineering (MLOps/DataOps) in contrast to DevOps. * Best practices for developing maintainable ML and Data applications. * How Flyte can be used to bridge these gaps for users with varying technical proficiency.