Exabyte-scale Streaming Iceberg IO with Beam, Ray, and DeltaCAT


Speaker(s):
Patrick Ames

Exabyte-scale Streaming Iceberg IO with Beam, Ray, and DeltaCAT

By Patrick Ames
Jul-8 09:45-10:15 in Horizon Hall
Add to Calendar 07/08/2025 9:45 AM 07/08/2025 10:15 AM BS25: Exabyte-scale Streaming Iceberg IO with Beam, Ray, and DeltaCAT

Production case study highlighting how Amazon uses Ray and DeltaCAT at exabyte-scale to resolve longstanding performance & scale challenges integrating streaming pipelines with Apache Iceberg. Highlights how the Apache Beam, Ray, Apache Flink, and Apache Spark communities can start bringing the same benefits to their workloads using the DeltaCAT project’s IO source/sink implementations for Apache Beam.

Horizon Hall

Production case study highlighting how Amazon uses Ray and DeltaCAT at exabyte-scale to resolve longstanding performance & scale challenges integrating streaming pipelines with Apache Iceberg. Highlights how the Apache Beam, Ray, Apache Flink, and Apache Spark communities can start bringing the same benefits to their workloads using the DeltaCAT project’s IO source/sink implementations for Apache Beam.