Autoscaling your transforms with auto-sharded GroupIntoBatches

Speaker(s):

Autoscaling your transforms with auto-sharded GroupIntoBatches

By Pablo Estrada

Aug-5 21:30-21:55

Add to Calendar 08/05/2021 9:30 PM 08/05/2021 9:55 PM America/Los_Angeles AS24: Autoscaling your transforms with auto-sharded GroupIntoBatches

Big data systems have implemented the ability to scale up from the cluster perspective: Add more workers, and parallelize further.

An issue with this is that many transforms still work with a fixed number of shards that can’t scale up or down with the cluster.

We recently added this capability to Cloud Dataflow, and we want to share our experiments and what we learned from this - and how you can take advantage of it.

Big data systems have implemented the ability to scale up from the cluster perspective: Add more workers, and parallelize further.

An issue with this is that many transforms still work with a fixed number of shards that can’t scale up or down with the cluster.

We recently added this capability to Cloud Dataflow, and we want to share our experiments and what we learned from this - and how you can take advantage of it.