Scaling Real-Time Feature Generation Platform @Lyft

At Lyft, real-time feature generation is crucial for powering many business critical use-cases. This session describes how we leveraged Apache Beam to build a robust and scalable real-time feature generation platform for this purpose, capable of generating 100s of millions of features per minute. We will delve into the critical factors that engineering teams should consider when designing a real-time feature generation platform, such as: Data consistency and accuracy, with a focus on ownership and quality guarantees. Latency requirements Performance optimization to ensure efficient feature serving. Feature serving and downstream model execution pipelines. Data lineage tools for improved traceability. Strategies for designing for performance and minimizing infrastructure costs. The presentation will discuss engineering challenges encountered while scaling the Beam pipeline to support our requirements and the lessons we learned along the way.