Adding OpenTelemetry to Production Apps: Lessons Learned

The presentation discusses the challenges of implementing OpenTelemetry in a microservices architecture and the importance of monitoring and observability in modern applications.

• OpenTelemetry is a tool used for monitoring and observability in modern applications
• Implementing OpenTelemetry in a microservices architecture can be challenging
• Microservices create complex systems that require debugging, monitoring, and observability
• The scale of data in modern applications is massive and requires efficient logging and tracing
• Clock skew and distributed timing can also be problematic in distributed environments

The speaker mentions that the application they built, Bank of Sirius, is open source and reflects a real-world microservices architecture. They also emphasize the importance of using open source tools and products in modern applications.

Observability is increasingly important in our modern apps/cloud-native world. However, when adding observability to existing production apps, there are a number of tradeoffs in approaches and in tools. Often, these tradeoffs are an exercise in confusion, leading to decision paralysis. We took on the challenge of adding observability to NGINX MARA, investigating choices, discovering and addressing challenges while keeping to open source solutions whenever possible. You'll come away with an understanding of how the three classes of data (Metrics, Traces, Logs) work together, why we chose the solutions we used and how we extended past the normal space into health checks, introspection and core dumps. Come learn from our experience in dealing with OpenTelemetry and related tools, from traces, metrics and logs, in working with production class apps and discover what approach finally worked for us.