Project Shield: How we use Beam to defend democracy and free expression, and how we got started!

Project Shield is a service that counters distributed-denial-of-service (DDoS) attacks. It is available free of charge to eligible websites that have news, elections, and human rights related content. Project Shield helps ensure unhindered access to election-related information during global democratic processes (such as the U.S. 2022 midterm election season and many others). It also enables critical infrastructure and news websites to defend against non-stop attacks and provides crucial services and information during crises (such as the invasion of Ukraine).

In this keynote, Marc Howard will explain why and how Project Shield uses Apache Beam and Google Cloud Dataflow to deliver some of their core value. Their streaming Apache Beam pipelines process more than 3 TB of log data daily at significantly over 10,000 queries per second, growing significantly during large attacks of up to 400 million queries per second. These metrics power user-facing graphs and long-term attack analytics at scale, fine-tuning Project Shield’s defenses and supporting them in the effort of making the web a safe and free space.