PyPSA / PyPSA/linopy

Opt-in allocator arena release after Model.solve() to bound RSS across sequential solves

Open
#720 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement performance
Dominant language
Python
Stars
257
Forks
87
Avg merge
1d 3h
Merged PRs (30d)
29

Description

Summary

Sequential PyPSA-Eur-style chunked solves (weekly LPs, identical structure, N iterations in one process) exhibit monotonic RSS growth until OOM, even after del model + gc.collect() between iterations. Root cause is platform allocator arena retention (glibc on Linux, libSystem on macOS), not Python-level leaks. Proposing a small opt-in helper that releases arena pages back to the kernel after the existing solve path completes.

This is orthogonal to:

  • #630 (CSR freeze=True) — reduces peak within a single solve.
  • #699 (persistent solver / ModelDiff) — avoids rebuilds between solves.
  • #219 — NetCDF round-trip to lower peak during a single solve (has an unresolved SOS-undo round-trip gap, see discussion there).

Neither addresses the allocator-arena pattern: glibc / libSystem keep freed pages cached, so RSS grows monotonically across the chunk loop even when the Python-side working set is bounded.

Proposal

linopy/_memory.py (~30 LOC):

def release_allocator_pages() -> None:
    """gc.collect() then ask platform allocator to return arenas to kernel.

    Linux:   libc.malloc_trim(0)
    macOS:   libSystem.malloc_zone_pressure_relief(NULL, 0)
    Other:   gc only, silent no-op.
    """

Wired into Model.solve(release_memory: bool = False). Default unchanged. Also exposed at top level (linopy.release_allocator_pages()) so callers in chunked-solve loops can trim mid-pipeline without going through Model.solve.

Benchmark (macOS arm64, linopy master 37af4ba)

5 sequential solves of an identical 5k var × 2k con dense-block LP via HiGHS, single Python process:

metric baseline release_memory=True Δ
RSS post-cleanup, iter 4 (GB) 4.41 2.02 −54 %
ru_maxrss (GB) 5.30 4.32 −19 %
wall-clock total (s) 30.97 32.09 +3.6 % (noise)

Baseline RSS climbs 3.71 → 4.41 GB monotonically across the 5 iterations. With trim, post-cleanup RSS drops back to 1.5–2.0 GB after each solve. Linux numbers via malloc_trim typically more dramatic — happy to add a Linux benchmark before opening the PR.

Tests
  • Platform-mocked dispatch (Linux / Darwin / Windows / FreeBSD).
  • Kwarg default backward-compat smoke test.
  • Repeated-call stability (no crashes if libc / libSystem lookup misses).
  • Swallows OSError / AttributeError, debug-logs on miss.
Open questions
  1. Naming: release_memory=True on Model.solve vs. top-level linopy.release_allocator_pages() vs. both. Leaning toward both — kwarg for the common single-solve case, public helper for chunked loops where users want to trim between solves without re-entering Model.solve.
  2. Interaction with #699: once persistent solver lands, native solver model + factor stay resident. Python-side arena cache still grows; trim hook stays useful. Agree?
  3. Should the helper also clear m._xCounter / m._cCounter style state on Model.solve(release_memory=True)? Leaning no — out of scope for an allocator helper, separate concern.

Will open PR once direction is confirmed.

Refs: #219, #630, #699

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with linopy/_memory.py and the existing Model.solve path. Review the platform-mocked dispatch, backward-compatibility, repeated-call, and allocator-lookup failure tests described in the issue. Done means an opt-in release path is exposed as agreed, defaults remain unchanged, supported platforms are handled safely, and the proposed tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.