Case study: Using statefulDofns to process late arriving data

Jun-14 14:30-14:55 UTC
Room: Palisades

We needed to process two different types of files arriving in the same bucket but there was no way of knowing if both files had arrived in real time. So we used two separate beam pipelines and StatefulDoFns to wait until the all the files are received and processed.