Case study: Using statefulDofns to process late arriving data


Speaker(s):
Amruta Deshmukh

Case study: Using statefulDofns to process late arriving data

By Amruta Deshmukh
Jun-14 14:30-14:55 in Palisades
Add to Calendar 06/14/2023 2:30 PM 06/14/2023 2:55 PM America/Los_Angeles AS24: Case study: Using statefulDofns to process late arriving data

We needed to process two different types of files arriving in the same bucket but there was no way of knowing if both files had arrived in real time. So we used two separate beam pipelines and StatefulDoFns to wait until the all the files are received and processed.

 Palisades

We needed to process two different types of files arriving in the same bucket but there was no way of knowing if both files had arrived in real time. So we used two separate beam pipelines and StatefulDoFns to wait until the all the files are received and processed.