Cost Effective Solutions for Beam pipelines in Dataflow


Speaker(s):
Sharan Teja Malyala

Cost Effective Solutions for Beam pipelines in Dataflow

By Sharan Teja Malyala

  Download slides

This talk will explain some of the capabilities of Dataflow that can help users save costs:

  1. Dynamic Thread Scaling: Dataflow tries to maximize the worker utilisation with this feature and avoids using up more workers.
  2. Right fitting: Customize the resources at stage level and efficiently use the worker resources.
  3. Streaming autoscaling: Tune autoscaling behavior to save costs.