Using Apache Beam to power and scale a data engineering transformation at a Financial Exchange


Speaker(s):
Conall Bennett

Using Apache Beam to power and scale a data engineering transformation at a Financial Exchange

By Conall Bennett
Jul-8 10:45-11:10 in Horizon Hall
Add to Calendar 07/08/2025 10:45 AM 07/08/2025 11:10 AM BS25: Using Apache Beam to power and scale a data engineering transformation at a Financial Exchange

At CME Group we are going through a multi-year transformation to run the world’s leading derivatives exchange on Google’s cloud platform.

This talk will focus on how we have been using dataflow & apache beam as a core technology service to enable & transform how distributed data engineering teams across different business areas collaborate, build and deliver data products for internal stakeholders and customers, at scale and with a much faster time to market.

We’ll cover how we have adapted beam for stateful processing pipelines on key products requiring highly available, gapless, ordered & low latency inputs for unbounded, high volume streams of financial market data. We’ll talk about dataflow interoperability features we’re exploring for integration with quantitative modelling libraries and thoughts on democratising dataflow pipeline creation for non-engineers.

Horizon Hall

At CME Group we are going through a multi-year transformation to run the world’s leading derivatives exchange on Google’s cloud platform.

This talk will focus on how we have been using dataflow & apache beam as a core technology service to enable & transform how distributed data engineering teams across different business areas collaborate, build and deliver data products for internal stakeholders and customers, at scale and with a much faster time to market.

We’ll cover how we have adapted beam for stateful processing pipelines on key products requiring highly available, gapless, ordered & low latency inputs for unbounded, high volume streams of financial market data. We’ll talk about dataflow interoperability features we’re exploring for integration with quantitative modelling libraries and thoughts on democratising dataflow pipeline creation for non-engineers.