google / google/go-jsonnet

apparent deadlock in Python bindings for go-jsonnet

Open
#484 10 comments 2 reactions 0 assignees View on GitHub
bug help wanted Python support
Dominant language
Go
Stars
1.8k
Forks
263
PR merge metrics
No merged PRs in 30d

Description

First off - apologies for lack of any "simple reproduction" of this insidious bug - it only occurs under a specific set of (rather complex) circumstances. I will do my best to describe them and what I've observed - perhaps that will be enough to inspire a possible cause/solution.

Issue: I observe what appears to be a deadlock when calling _gojsonnet.evaluate_file() under a certain set of conditions. There is 0% CPU usage and the process appears to be hanging on IO somewhere in the call to evaluate_file().

My observations so far

- This bug is deterministic. When the "conditions" are met it seems to happen 100% of the time.

- It only affects gojsonnet bindings (v0.16 and v0.17 ... possibly earlier too, but these are the only ones I've been able to easily install). In my testing it **does not** affect C++ jsonnet bindings.

- I've only been able to trigger it when calling via PythonOperator on Apache Airflow (using LocalExecutor). In my setup, Airflow is running as a Docker container (hopefully not relevant but worth mentioning). I briefly dug into the PythonOperator code and it appears to launch a new Python subprocess in case that's important. I have not been able to trigger it by calling evaluate_file() manually from the Python REPL (verified both locally and inside my Docker container environment).

- It seems to only happen when the generated json is rather large (~5MB in my case). If I shrink the size of my generated json to a few KB the issue goes away. I don't know exactly where this cutoff is, but the deterministic nature of the bug strongly suggests there is a hard cutoff somewhere.

Please let me know what you think or if you have any questions - I don't have enough knowledge of how Python's C API works to really debug this further on my own. There are unfortunately lots of moving parts here, but I filed as an issue in go-jsonnet given that the C++ bindings do not exhibit this issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.