[BUG] memory leak and crash?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 18k
- Forks
- 2.3k
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 10
Description
Issue description
I am facing memory leak and crash issues in pybind11.
I am calling a python function "myfunc" from a python file "mydl.py" that uses Tensorflow Keras deep learning functions, Numpy, and Redis modules using pybind11 in a repeatitive C++ code.
The example code is as follows.
Reproducible example code
class myclass {
public:
myclass() {
py::initialize_interpreter();
{
py::module sys = py::module::import("sys");
py::module os = py::module::import("os");
py::str cwd = os.attr("getcwd")();
py::print("os.cwd: ", cwd);
py::str bin = cwd + py::str("/../bin");
// Add bin to sys.path
py::module site = py::module::import("site");
site.attr("addsitedir")(bin);
}
}
~myclass() {
py::finalize_interpreter();
}
int callpyfunc(string a1, string a2) {
int retval;
{
py::module mydl = py::module::import("mydl");
py::object result = mydl.attr("myfunc")(a1, a2);
retval = result.cast<int>();
}
return retval;
}
}
myclass *mcobj1;
int main() {
mcobj1 = new myclass();
int retval;
while (/* some deep learning condition is not met */) {
retval = mcobj1->callpyfunc(a1, a2);
}
del mcobj1;
}
Outcome
The memory size of this program goes on increasing consistently to the point of it consuming entire 62 GB RAM and crashing. It seems like Python interpreter is not releasing memory allocated for different objects inside each call to "myfunc" of "mydl.py" even after the call gets done.
Here's what all I have tried with no luck of fixing the issue:
-
Using scoped interpreter inside
callpyfuncinstead of doinginitialize_interpreterandfinalize_interpreter. But in that case the code crashes quietly in the second call to "callpyfunc", the first call goes fine. This is exactly what is mentioned here. -
Moving
initialize_interpreteralong with import of modules like "sys", "os" andfinalize_interpreterinsidecallpyfunc. But in that case the code crashes in the second call to "callpyfunc" at linepy::module mydl = py::module::import("mydl");and never reaches finalizing of interpreter.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the repeated callpyfunc path in the provided example, especially py::module::import("mydl"), initialize_interpreter(), and finalize_interpreter(), then compare the lifecycle behavior with issue #1439. Reproduce the second-call crash and memory growth using a minimal mydl.py case before involving TensorFlow Keras, Numpy, or Redis; done means identifying whether the behavior belongs to pybind11 or an imported dependency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python, redis
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100