Opt-in allocator arena release after Model.solve() to bound RSS across sequential solves
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 257
- Forks
- 87
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 29
Description
Summary
Sequential PyPSA-Eur-style chunked solves (weekly LPs, identical structure, N iterations in one process) exhibit monotonic RSS growth until OOM, even after del model + gc.collect() between iterations. Root cause is platform allocator arena retention (glibc on Linux, libSystem on macOS), not Python-level leaks. Proposing a small opt-in helper that releases arena pages back to the kernel after the existing solve path completes.
This is orthogonal to:
- #630 (CSR
freeze=True) — reduces peak within a single solve. - #699 (persistent solver /
ModelDiff) — avoids rebuilds between solves. - #219 — NetCDF round-trip to lower peak during a single solve (has an unresolved SOS-undo round-trip gap, see discussion there).
Neither addresses the allocator-arena pattern: glibc / libSystem keep freed pages cached, so RSS grows monotonically across the chunk loop even when the Python-side working set is bounded.
Proposal
linopy/_memory.py (~30 LOC):
def release_allocator_pages() -> None:
"""gc.collect() then ask platform allocator to return arenas to kernel.
Linux: libc.malloc_trim(0)
macOS: libSystem.malloc_zone_pressure_relief(NULL, 0)
Other: gc only, silent no-op.
"""
Wired into Model.solve(release_memory: bool = False). Default unchanged. Also exposed at top level (linopy.release_allocator_pages()) so callers in chunked-solve loops can trim mid-pipeline without going through Model.solve.
Benchmark (macOS arm64, linopy master 37af4ba)
5 sequential solves of an identical 5k var × 2k con dense-block LP via HiGHS, single Python process:
| metric | baseline | release_memory=True |
Δ |
|---|---|---|---|
| RSS post-cleanup, iter 4 (GB) | 4.41 | 2.02 | −54 % |
ru_maxrss (GB) |
5.30 | 4.32 | −19 % |
| wall-clock total (s) | 30.97 | 32.09 | +3.6 % (noise) |
Baseline RSS climbs 3.71 → 4.41 GB monotonically across the 5 iterations. With trim, post-cleanup RSS drops back to 1.5–2.0 GB after each solve. Linux numbers via malloc_trim typically more dramatic — happy to add a Linux benchmark before opening the PR.
Tests
- Platform-mocked dispatch (Linux / Darwin / Windows / FreeBSD).
- Kwarg default backward-compat smoke test.
- Repeated-call stability (no crashes if
libc/libSystemlookup misses). - Swallows
OSError/AttributeError, debug-logs on miss.
Open questions
- Naming:
release_memory=TrueonModel.solvevs. top-levellinopy.release_allocator_pages()vs. both. Leaning toward both — kwarg for the common single-solve case, public helper for chunked loops where users want to trim between solves without re-enteringModel.solve. - Interaction with #699: once persistent solver lands, native solver model + factor stay resident. Python-side arena cache still grows; trim hook stays useful. Agree?
- Should the helper also clear
m._xCounter/m._cCounterstyle state onModel.solve(release_memory=True)? Leaning no — out of scope for an allocator helper, separate concern.
Will open PR once direction is confirmed.
Refs: #219, #630, #699
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with linopy/_memory.py and the existing Model.solve path. Review the platform-mocked dispatch, backward-compatibility, repeated-call, and allocator-lookup failure tests described in the issue. Done means an opt-in release path is exposed as agreed, defaults remain unchanged, supported platforms are handled safely, and the proposed tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100