See the Full Picture: Integrating Beam/Dataflow into Your Distributed Traces

Achieving holistic observability often hits a wall at asynchronous batch or streaming systems. While OpenTelemetry provides standards for tracing, integrating systems like Apache Beam/Dataflow requires specific considerations. This presentation details the successful integration of Beam pipelines with OpenTelemetry’s tracing APIs. We’ll explore the mechanisms for context propagation across Beam’s distributed workers and stages, enabling pipelines to join traces initiated by upstream services. Discover how spans generated within the Dataflow runner can be exported and visualized alongside the rest of your application traces in Google Cloud Trace, finally delivering the “full picture” of your system’s behavior, including its data processing components.