pybind / pybind/pybind11

[BUG] memory leak and crash?

Open
#2,929 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Issue description

I am facing memory leak and crash issues in pybind11.

I am calling a python function "myfunc" from a python file "mydl.py" that uses Tensorflow Keras deep learning functions, Numpy, and Redis modules using pybind11 in a repeatitive C++ code.

The example code is as follows.

Reproducible example code

class myclass {
  public:
   myclass() {
     py::initialize_interpreter();

     {
        py::module sys = py::module::import("sys");
        py::module os = py::module::import("os");
        py::str cwd = os.attr("getcwd")();
        py::print("os.cwd: ", cwd);
        py::str bin = cwd + py::str("/../bin");

        // Add bin to sys.path
        py::module site = py::module::import("site");
        site.attr("addsitedir")(bin);
      }
   }
   
   ~myclass() {
      py::finalize_interpreter();
   }

   int callpyfunc(string a1, string a2) {
     int retval;
     {
        py::module mydl = py::module::import("mydl");
        py::object result = mydl.attr("myfunc")(a1, a2);
        retval = result.cast<int>();
     }

    return retval;
   }
}

myclass *mcobj1;

int main() {

  mcobj1 = new myclass();
  int retval;

  while (/* some deep learning condition is not met */) {
     retval = mcobj1->callpyfunc(a1, a2);
  }

  del mcobj1;

}

Outcome

The memory size of this program goes on increasing consistently to the point of it consuming entire 62 GB RAM and crashing. It seems like Python interpreter is not releasing memory allocated for different objects inside each call to "myfunc" of "mydl.py" even after the call gets done.

Here's what all I have tried with no luck of fixing the issue:

  1. Using scoped interpreter inside callpyfunc instead of doing initialize_interpreter and finalize_interpreter. But in that case the code crashes quietly in the second call to "callpyfunc", the first call goes fine. This is exactly what is mentioned here.

  2. Moving initialize_interpreter along with import of modules like "sys", "os" and finalize_interpreter inside callpyfunc. But in that case the code crashes in the second call to "callpyfunc" at line py::module mydl = py::module::import("mydl"); and never reaches finalizing of interpreter.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the repeated callpyfunc path in the provided example, especially py::module::import("mydl"), initialize_interpreter(), and finalize_interpreter(), then compare the lifecycle behavior with issue #1439. Reproduce the second-call crash and memory growth using a minimal mydl.py case before involving TensorFlow Keras, Numpy, or Redis; done means identifying whether the behavior belongs to pybind11 or an imported dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, redis
Domain
devtools
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.