Troubleshooting Python pipelines with process monitoring tools.

In an ideal scenario, a data processing pipeline performs without issues. When a runtime processing error occurs, normally Beam surfaces the error to the runner. However in some cases, the process running the user code might run out of memory, get stuck or crash. This can prevent it from reporting the error, leaving the user unaware of the failure’s root cause. In this talk, I’ll discuss troubleshooting techniques for these situations. The techniques I cover can also be applied for debugging other Python applications.