Bridging BigQuery and ClickHouse with Apache Beam: A Google Dataflow Template for Batch Ingestion


Speaker(s):
Bentsi Leviav

Bridging BigQuery and ClickHouse with Apache Beam: A Google Dataflow Template for Batch Ingestion

By Bentsi Leviav
Jul-8 15:00-15:25 in Star Leaved Gum
Add to Calendar 07/08/2025 3:00 PM 07/08/2025 3:25 PM America/New_York BS25: Bridging BigQuery and ClickHouse with Apache Beam: A Google Dataflow Template for Batch Ingestion

In this talk, I’ll walk through the process of developing and deploying a reusable Dataflow template using ClickHouseIO, the official Beam connector. The template enables users to duplicate and copy data from BigQuery to ClickHouse - a high-performance OLAP database increasingly adopted for real-time analytics.

In addition, the session will cover key considerations when working with the ClickHouseIO connector such as schema mapping, handling large volumes of data, performance tuning, and more.

A live demo will showcase a real-world use case of duplicating data from BigQuery to ClickHouse using the template, demonstrating how Apache Beam is the foundation to power this cross-system integration.

Star Leaved Gum

In this talk, I’ll walk through the process of developing and deploying a reusable Dataflow template using ClickHouseIO, the official Beam connector. The template enables users to duplicate and copy data from BigQuery to ClickHouse - a high-performance OLAP database increasingly adopted for real-time analytics.

In addition, the session will cover key considerations when working with the ClickHouseIO connector such as schema mapping, handling large volumes of data, performance tuning, and more.

A live demo will showcase a real-world use case of duplicating data from BigQuery to ClickHouse using the template, demonstrating how Apache Beam is the foundation to power this cross-system integration.