apparent deadlock in Python bindings for go-jsonnet
- Dominant language
- Go
- Stars
- 1.8k
- Forks
- 263
- PR merge metrics
- No merged PRs in 30d
Description
First off - apologies for lack of any "simple reproduction" of this insidious bug - it only occurs under a specific set of (rather complex) circumstances. I will do my best to describe them and what I've observed - perhaps that will be enough to inspire a possible cause/solution.
Issue: I observe what appears to be a deadlock when calling _gojsonnet.evaluate_file() under a certain set of conditions. There is 0% CPU usage and the process appears to be hanging on IO somewhere in the call to evaluate_file().
My observations so far
- This bug is deterministic. When the "conditions" are met it seems to happen 100% of the time.
- It only affects gojsonnet bindings (v0.16 and v0.17 ... possibly earlier too, but these are the only ones I've been able to easily install). In my testing it **does not** affect C++ jsonnet bindings.
- I've only been able to trigger it when calling via PythonOperator on Apache Airflow (using LocalExecutor). In my setup, Airflow is running as a Docker container (hopefully not relevant but worth mentioning). I briefly dug into the PythonOperator code and it appears to launch a new Python subprocess in case that's important. I have not been able to trigger it by calling evaluate_file() manually from the Python REPL (verified both locally and inside my Docker container environment).
- It seems to only happen when the generated json is rather large (~5MB in my case). If I shrink the size of my generated json to a few KB the issue goes away. I don't know exactly where this cutoff is, but the deterministic nature of the bug strongly suggests there is a hard cutoff somewhere.
Please let me know what you think or if you have any questions - I don't have enough knowledge of how Python's C API works to really debug this further on my own. There are unfortunately lots of moving parts here, but I filed as an issue in go-jsonnet given that the C++ bindings do not exhibit this issue.
Contributor guide
Assessment
This issue has not been assessed yet.