C Unpickler memory leak when unpickling big geojsons
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug report
Bug description:
I noticed that the pickle.load method leaves more memory allocated than just the size of the returned object when unpickling geojson dictionaries.
I initially noticed the leak in my docker container running on x86. Here is a code snippet that reproduced the problem for me on M1 MacOS Sonoma (python 3.11.9).
import io
import pickle
import psutil
process = psutil.Process()
to_pickle = {
"type": "FeatureCollection",
"crs": {"properties": {"name": "EPSG:4326"}, "type": "name"},
"features": [
{
"type": "Feature",
"id": f"id{idx}",
"properties": {
"str_prop": f"str_var{idx}",
"int_prop": idx,
"bool_prop": bool(idx % 2),
"none_prop": None,
"float_prop": float(idx),
},
}
for idx in range(9000)
],
}
b = pickle.dumps(to_pickle)
del to_pickle
print(
f"Initial memory usage: {round(int(process.memory_info().rss / 1024) / 1024, 3)}MiB"
)
pickle.Unpickler(io.BytesIO(b)).load()
print(
f"After 1st unpickle memory usage: {round(int(process.memory_info().rss / 1024) / 1024, 3)}MiB"
)
for i in range(1000):
pickle.Unpickler(io.BytesIO(b)).load()
print(
f"Final memory usage: {round(int(process.memory_info().rss / 1024) / 1024, 3)}MiB"
)
CPython versions tested on:
3.11
Operating systems tested on:
Linux, macOS
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Python reproducer on the affected CPython versions and compare memory after repeated pickle.Unpickler(...).load() calls. Trace the C Unpickler path used for these nested dictionaries; done means repeated unpickling no longer leaves the reported excess memory allocated, with a regression test added for the case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100