Scaling Public Internet Data Collection With Apache Beam


Speaker(s):
Lior Dadosh

Scaling Public Internet Data Collection With Apache Beam

By Lior Dadosh
Jun-13 11:30-11:55 in Horizon
Add to Calendar 06/13/2023 11:30 AM 06/13/2023 11:55 AM America/Los_Angeles AS24: Scaling Public Internet Data Collection With Apache Beam

In Cortex Xpanse, we are scanning the internet and collecting public internet data in order to identify issues in customers’ assets.

In this session, I’ll present the wide usage of Beam in our org (hint: we have over 1000 daily running jobs) and share our best practices. Be ready to learn how are we deploying, monitoring, optimizing, and testing our pipelines.

 Horizon

  Download slides

In Cortex Xpanse, we are scanning the internet and collecting public internet data in order to identify issues in customers’ assets.

In this session, I’ll present the wide usage of Beam in our org (hint: we have over 1000 daily running jobs) and share our best practices. Be ready to learn how are we deploying, monitoring, optimizing, and testing our pipelines.