Leveraging LLMs for Agentic Workflow Orchestration in Apache Beam YAML Pipelines

This session explores how Large Language Models (LLMs) can be integrated into Apache Beam tooling to enable agentic orchestration of YAML-defined workflows. We present a system where LLMs parse, validate, and execute Beam YAML pipelines, acting as autonomous agents that enhance workflow automation and reduce manual intervention. The talk covers architecture, pipeline translation, task planning, and integration strategies for embedding LLMs in declarative workflow environments. Attendees will learn how to build intelligent tooling layers for Beam that support dynamic pipeline generation, error resolution, and adaptive execution—all while maintaining the flexibility and scalability of the Beam programming model.