Understand Systems with OpenTelemetry: A Hybrid Telemetry Data Backend

Open Gemini is a unified storage engine that processes data generated and collected by a simple data stack. It uses Open Telemetry as the standard protocol for visibility and provides a metric store, log store, and observability backend storage.

• Open Gemini is a unified storage engine that processes data generated and collected by a simple data stack.
• It uses Open Telemetry as the standard protocol for visibility and provides a metric store, log store, and observability backend storage.
• Open Telemetry provides a unified access protocol and standardized data models for metrics, trace, and logs.
• Open Gemini uses a columnar layout for the metric store and an airway index for the log store.
• Open Gemini provides a unified Circle engine for all queries.
• The sale rate index balances the linked list and reduces the total size of the index.
• Open Gemini reduces CPU cost and improves query latency compared to other systems.
• Open Gemini aims to support tracing data storage and correlation analysis in the future.

Open Gemini replaced the original system and reduced CPU cost by 16%. However, when the tag value continuity reached 10 milliamp node, performance started to decrease. The team had to tackle this problem to improve the system.

HUAWEI CLOUD uses the cloud-native architecture to support thousands of services, DevOps requires understanding the running status of each system in a large number of interdependent microservices, middleware, and devices. openTelemetry is an observability industry standard. It provides standards and tools to generate high-quality telemetry data (metrics, logs, and traces). However, considering the need to quickly understand the system running status in massive telemetry data, a back-end storage that hybrid multiple types of telemetry data is a key part of the observability system. How to support efficient correlation query and real-time analysis in massive high-cardinality telemetry data and reduce the cost of telemetry data storage and computing is a challenge for us. In the sharing, we will introduce: 1. Key Challenges to Cloud Native Observability of HUAWEI CLOUD. 2. From metric data to telemetry data, the evolution history and thinking of observability back-end storage. 3. HUAWEI CLOUD observability cases.