Hot Key Detection and Handling in Apache Beam Pipelines

Jun-13 15:30-15:55 UTC
Room: Palisades

  • Overview of hot keys and their impact on Beam pipelines

  • Techniques for hot key detection and handling, including key partitioning and key normalization

  • Best practices for dealing with hot keys in Beam pipelines, such as using custom combiners and windowing, and tuning pipeline settings

  • Some examples of Dataflow pipelines where hot key detection and handling proved to be critical for successful pipeline execution.

Attendees will leave this talk with a deeper understanding of how to identify and address hot key issues in their Beam pipelines, leading to better pipeline performance, scalability, and reliability.