logo

Best Practices for Accelerated Image Distribution Using Dragonfly

2023-04-19

Authors:   Wenbo Qi, Yiyang Huang


Summary

Best practices for using Dragonfly to accelerate image distribution in cloud services
  • Dragonfly is a P2P-based image and file distribution system used to improve the physical and speed of large-scale field distribution
  • Dragonfly has been selected and put into production by many internet companies since it was open-sourced in 2017
  • The system architecture of Dragonfly consists of four services: manager, scheduler, asset peer, and pure peer
  • The manager service is used to manage the relationship between multi-clusters or P2P clusters and provides a dynamic configure management
  • The scheduler service selects candidate download parents for download peer when the pure download fails and controls the peer to download the task
  • The asset peer is triggered by the scheduler to download back to songs and divide the results into pieces
  • The pure peer is the calendar in P2P network and can be downloaded and uploaded
  • Dragonfly can be combined and used with other systems in the ecosystem, such as Harbor, Nydus, and eStargz
Dragonfly has been used by many internet companies since it was open-sourced in 2017. One example is the use of Dragonfly in Bytedance's cloud service Volcano Engine, where it is used for container image hosting services to manage the entire life cycle of container images. Another example is the use of Dragonfly in machine learning, where it has been used to download modules from object storage through P2P technology, providing efficient energy to download the modules and reducing the bandwidth on the object stories.

Abstract

This sharing introduces dragonfly, which is a P2P-based image and file distribution system. Introduce the system architecture of dragonfly and how to select the technology in the design. Provides best practices for image acceleration using dragonfly in company. Finally, describe how dragonfly can be combined and used with other systems in the ecosystem, such as Harbor, Nydus, eStargz, etc.

Materials: