llnl / llnl/units

python with pyinstrument seems to show a memory leak

Open
#368 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
171
Forks
35
Avg merge
2d 5h
Merged PRs (30d)
3

Description

Hi,
I am using Units from python. One thing that I have noticed is that, when I set up a test with pyinstrument @profile (https://github.com/joerick/pyinstrument), the inscances of Unit seems to be leaked. Here a minimal example:

import functools
from collections.abc import Callable
from typing import Any, TypeVar

from memory_profiler import profile
from pyinstrument import Profiler
from units_llnl import Unit

F = TypeVar("F", bound=Callable[..., Any])

PROFILED_FILES: list[str] = []


def profile_function(output_file: str) -> Callable[[F], F]:

    def decorator(func: F) -> F:
        @functools.wraps(func)
        def wrapper(*args: Any, **kwargs: Any) -> Any:
            profiler = Profiler(0.00001, async_mode="enabled")
            profiler.start()
            result = func(*args, **kwargs)  # Run the function
            profiler.stop()
            namefile = f"{output_file}.html"
            profiler.write_html(namefile)
            return result

        return wrapper  # type: ignore

    return decorator


@profile
@profile_function("eval")
def test_silly() -> None:
    class dummy:
        def __init__(self, theunit: str) -> None:
            self.unit = Unit(1, theunit)

    for ii in range(100):
        aa = dummy("m")
        bb = dummy("kg")

For such example i get in the terminal:

nanobind: leaked 2 instances!
nanobind: leaked 1 types!
  leaked type "units_llnl.units_llnl_ext.Unit"
nanobind: leaked 34 functions!
  leaked function "to_dict"
  leaked function "__mul__"
  leaked function "is_equation"
  leaked function "isinf"
  leaked function "__hash__"
  leaked function "__ne__"
  leaked function "__truediv__"
  leaked function "is_valid"
  leaked function "is_exactly_the_same"
  leaked function ""
  leaked function "is_convertible_to"
  ... skipped remainder
nanobind: this is likely caused by a reference counting issue in the binding code.

This actually seems a true leak, here the report from the memory_profiler:

46    263.5 MiB    263.5 MiB           1           @functools.wraps(func)
47                                                 def wrapper(*args: Any, **kwargs: Any) -> Any:
48    263.5 MiB      0.0 MiB           1               profiler = Profiler(0.00001, async_mode="enabled")
49    263.5 MiB      0.0 MiB           1               profiler.start()
50    266.6 MiB      3.1 MiB           1               result = func(*args, **kwargs)  # Run the function
51    266.6 MiB      0.0 MiB           1               profiler.stop()
52    266.6 MiB      0.0 MiB           1               namefile = f"{output_file}.html"
53    267.6 MiB      1.0 MiB           1               profiler.write_html(namefile)
54    267.6 MiB      0.0 MiB           1               return result

Thank you,
Mattia

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal Python reproducer using Unit, pyinstrument, and memory_profiler, then inspect the nanobind leak report for the units_llnl_ext.Unit type and functions. Reproduce the reported retained instances and determine whether the binding cleanup is responsible; done means the example no longer reports leaked Unit instances or binding objects.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.