python / python/cpython

C Unpickler memory leak when unpickling big geojsons

Open
#121,569 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

I noticed that the pickle.load method leaves more memory allocated than just the size of the returned object when unpickling geojson dictionaries.

I initially noticed the leak in my docker container running on x86. Here is a code snippet that reproduced the problem for me on M1 MacOS Sonoma (python 3.11.9).

import io
import pickle

import psutil

process = psutil.Process()

to_pickle = {
    "type": "FeatureCollection",
    "crs": {"properties": {"name": "EPSG:4326"}, "type": "name"},
    "features": [
        {
            "type": "Feature",
            "id": f"id{idx}",
            "properties": {
                "str_prop": f"str_var{idx}",
                "int_prop": idx,
                "bool_prop": bool(idx % 2),
                "none_prop": None,
                "float_prop": float(idx),
            },
        }
        for idx in range(9000)
    ],
}
b = pickle.dumps(to_pickle)
del to_pickle

print(
    f"Initial memory usage: {round(int(process.memory_info().rss / 1024) / 1024, 3)}MiB"
)
pickle.Unpickler(io.BytesIO(b)).load()
print(
    f"After 1st unpickle memory usage: {round(int(process.memory_info().rss / 1024) / 1024, 3)}MiB"
)
for i in range(1000):
    pickle.Unpickler(io.BytesIO(b)).load()

print(
    f"Final memory usage: {round(int(process.memory_info().rss / 1024) / 1024, 3)}MiB"
)
CPython versions tested on:

3.11

Operating systems tested on:

Linux, macOS

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Python reproducer on the affected CPython versions and compare memory after repeated pickle.Unpickler(...).load() calls. Trace the C Unpickler path used for these nested dictionaries; done means repeated unpickling no longer leaves the reported excess memory allocated, with a regression test added for the case.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.