Connected Observability Pipelines in the OpenTelemetry Collector

The OpenTelemetry Collector is now a more capable tool for processing telemetry due to the introduction of the Connectors framework.

• The Connectors framework allows for the creation of generalized systems for managing telemetry.
• Connectors can be used to replicate and merge data streams, apply sampling criteria, and reason about multiple data types in one place.
• The structure of pipelines and data streams in the OpenTelemetry Collector is governed by certain rules and expectations.
• The Connectors framework can be used to address limitations in the existing pipeline structure.
• An anecdote is provided to illustrate how the Connectors framework can be used to filter and redact telemetry data.

In a scenario where an expensive analytics tool is unable to process all log data due to cost constraints, the Connectors framework can be used to replicate the data stream and filter one stream to send to the expensive tool while the other stream is sent to cheaper cold storage. Additionally, if there is a requirement to redact PII immediately upon ingestion, the Connectors framework can be used to replicate the data stream, redact the PII in one stream, and process the other stream.

The OpenTelemetry Collector is a powerful tool for collecting, processing, and forwarding observability signals. User-defined pipelines control the flow of data through the collector. A new feature has made it possible to link pipelines together, enabling more sophisticated control and processing of observability signals. In this session Dan Jaglowski will introduce connectors, a new type of pipeline component that directly links pipelines to one another. Connectors can replicate or route data between pipelines, reduce data volume with multi-signal logic (e.g. derive summary metrics from logs or spans), even detect and report anomalies. Come broaden your understanding of pipelines and learn how to implement multi-pipeline data flow strategies. If you're not already familiar with pipelines, don't worry. Dan will provide a concise introduction to the OpenTelemetry Collector and its pipeline framework before explaining how connectors extend the architecture.