Program 2026

As the Waymo fleet scales in size and across geos, low-latency observability is increasingly critical for providing our riders a safe and reliable experience. Stateful time-series processing built on Dataflow is a fundamental component of this observability stack, granting us a bird’s eye view of the fleet as our vehicles move through the world to serve our customers.

In this talk, we’ll investigate how Waymo leverages Dataflow to power our near-real-time use cases such as Fleet Monitoring, Importance Sampling, and more. We’ll also take a look at how Waymo is taking advantage of Google’s TPU fleet and LLMs to introduce streaming inference into the observability equation.

A fully autonomous VTOL logistics fleet has been flying mathematically optimized delivery corridors for 14 months. Every flight clears health checks. Every autopilot recovery is clean. No alerts fire. No reports are filed. On paper the operation is running perfectly.

A single Apache Beam pipeline replaying 1,200 archived flight logs tells a different story. Attitude recovery events — small, clean, individually insignificant — cluster at one specific waypoint, one altitude band, one azimuth range. The autopilot has been silently fighting a terrain-induced atmospheric rotor on every affected flight. It always won. It never complained. But the cumulative cost is real — excess battery draw, elevated motor wear, compounding flight time losses that no dashboard ever surfaced.

This talk shows how Beam’s unified model runs the same pipeline against 14 months of archived logs in batch and against the live telemetry stream in real time — no rewrite, no reconciliation, one codebase. We build it live, extend it layer by layer from raw telemetry through operator commands to LLM-generated cost analysis and route recommendations. The route gets adjusted. The inefficiency disappears.

The question Beam lets you ask for the first time: what has your fleet been quietly compensating for that your dashboards never showed you?

While Beam websites listed 30+ built-in IO connectors supporting streaming, the status (resilience, scalability, performance, etc) of each IO connectors is not equal. This session highlights recent improvements for selected Beam Streaming IO connectors that have indicated gaps between user demand and status quo in Beam, including Debezium, Jms, Mqtt, and Pulsar. We discusses how each connector went from “Day 1” existence to “Day 2” resilience, and with remarks on facilitating community engagements on Beam IO Ecosystem.

Traditionally, converting a Parquet-based data lake to Iceberg required a hidden tax of rewriting every single data file. For organizations managing petabyte-scale datasets, this compute overhead and the associated cloud bill are often dealbreakers.

This talk introduces a more efficient path using Apache Beam’s new AddFiles feature to perform zero-copy migrations, registering existing Parquet files directly into an Iceberg table without moving a single byte.

In this session, we’ll explore:

• A practical framework for modernizing your lakehouse with minimal compute overhead.
• Live demos showcasing (1) the Batch approach for migrating historical data and (2) the Streaming approach for registering incoming files in real-time
• A decision matrix for choosing between tradition rewrites and zero-copy registration

Scaling AI inference across thousands of workers to maximize throughput is a flagship feature of Apache Beam. However, this massive parallelism often collides head-on with strict external API quotas (e.g., Vertex AI, OpenAI).

To bridge this gap, we’ve introduced a Proactive Global RateLimiter for Apache Beam. Integrated directly into the RunInference transform and also made it available for custom DoFn’s. It moves quota management from reactive retry storms to proactive pacing.

This talk will explore how Beam coordinates rate limits across dispersed workers and communicates dynamic back pressure to the Runner Autoscaler to prevent compute waste. Attendees can expect to come away with an understanding of how global rate limiting works in distributed environments, how the autoscaler responds to rate signals, and how they can use Beam to scale their usecases safely without overwhelming external services.

This session explores recent innovations that enhance pipeline scalability, reliability, and availability. We will cover key updates in autoscaling, high availability, and reliability, alongside progress in streaming ML and IO excellence. Attendees will discover how these enhancements facilitate the building of robust, next-generation streaming architectures.

Time series data is a foundational and ubiquitous format in modern big data applications, driving insights in fields ranging from user activity tracking to IoT sensor monitoring. While there is growing interest in processing time series data within Apache Beam, its inherently unordered and parallel execution model forces developers to implement complex custom logic to handle chronological events accurately.

In this talk, we explore the crucial first step of time series processing in Beam: buffering data in precise timestamp order to enable accurate downstream analysis. We will evaluate and compare various buffering approaches, weighing their trade-offs. Finally, we will demonstrate these concepts in action through a real-world anomaly detection use case utilizing the recently developed BigQuery CDC source.

Introducing a YAML-driven Dataflow Flex Template design enabling product teams to self-serve Spanner to BigQuery replication, supporting both current-state and append-only modes for analytics, downstream applications, and audit trails. This session focuses on streaming CDC(Change Data Capture) use cases.

In a federated deployment model where each product team owns and operates its own pipeline, redeployments emerged as a recurring risk event. None of Dataflow’s existing restart mechanisms work perfectly for our use cases, leaving every redeployment risks data loss or duplicate data.

A proper fix belongs in Dataflow and SpannerIO. In the meantime, an interim solution has kept redeployments routine in production, with no data loss or duplication.

This session covers the problem, the approach, and the tradeoffs that remain.