NEXMark-Beam: Your best companion for testing and benchmarking new core stream processing libraries

We will provide an overview of FlinkNDB, which is a new type of state backend for scalable and ultra-flexible stateful data streaming that we are developing as part of our master thesis at RISE Research Institute of Sweden. We will also present how we leveraged ApacheBeam’s capabilities to speed up the development of the new Flink state backend and showcase its performance using NEXMark benchmarks and failure-recovery scenarios.

Topics covered:

• FlinkNDB - Quick overview
• NEXMark and Apache Beam’s NEXMark implementation
• Setting up Flink runner state backend’s on Apache Beam
• How we used NEXMark queries to evaluate the functionality and performance of state backends
• Benchmarking results