Power Realtime Machine Learning Feature Engineering with Managed Beam at LinkedIn


Speaker(s):
David Shao
Yanan Hao

Power Realtime Machine Learning Feature Engineering with Managed Beam at LinkedIn

By David Shao Yanan Hao
Jun-14 14:30-14:55 in Horizon
Add to Calendar 06/14/2023 2:30 PM 06/14/2023 2:55 PM America/Los_Angeles AS24: Power Realtime Machine Learning Feature Engineering with Managed Beam at LinkedIn

ML models are applied in all the key products like Job Recommendation, Search, Feed, and Ads in LinkedIn, powered by thousands of features about entities like companies, job postings, and LinkedIn members from the Economic Graph. Preparing and managing features has been one of the most time-consuming parts of operating ML applications at scale. There is a growing demand for fresh “real-time” feature data, which is expected to have significant business impact by boosting ML models’ relevancy performance.

A scalable framework solution for Realtime ML Feature Engineering, Managed-beam Feature Platform is built with the following main features:

  • Cross-language and cross-platform compatibility: Java and Python Beam APIs are provided to AI engineers to author the real-time feature generation pipeline in their preferred language.

  • Portable and scalable: Beam’s portable API allows AI engineers to write code once and run it on any supported platform without any modification. This also allows for easy scalability as AI engineers can scale their processing capacity up or down by simply changing the underlying processing engine, especially for integrations with external resources and MLOps services.

  • Managed Solution: Managed Beam Platform intelligently triages and mitigates operational issues in real-time through auto-sizing and auto-triaging. This allows for fully managed end-to-end operations by the platform with zero operational costs to ML users

Overall, Managed-beam Feature Platform empowers Realtime Machine Learning Feature Engineering with high usability, productivity and scalability.

 Horizon

ML models are applied in all the key products like Job Recommendation, Search, Feed, and Ads in LinkedIn, powered by thousands of features about entities like companies, job postings, and LinkedIn members from the Economic Graph. Preparing and managing features has been one of the most time-consuming parts of operating ML applications at scale. There is a growing demand for fresh “real-time” feature data, which is expected to have significant business impact by boosting ML models’ relevancy performance.

A scalable framework solution for Realtime ML Feature Engineering, Managed-beam Feature Platform is built with the following main features:

  • Cross-language and cross-platform compatibility: Java and Python Beam APIs are provided to AI engineers to author the real-time feature generation pipeline in their preferred language.

  • Portable and scalable: Beam’s portable API allows AI engineers to write code once and run it on any supported platform without any modification. This also allows for easy scalability as AI engineers can scale their processing capacity up or down by simply changing the underlying processing engine, especially for integrations with external resources and MLOps services.

  • Managed Solution: Managed Beam Platform intelligently triages and mitigates operational issues in real-time through auto-sizing and auto-triaging. This allows for fully managed end-to-end operations by the platform with zero operational costs to ML users

Overall, Managed-beam Feature Platform empowers Realtime Machine Learning Feature Engineering with high usability, productivity and scalability.