PyPSA / PyPSA/linopy

Zero culling forces model rebuilding

Open
#925 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug data-model performance
Dominant language
Python
Stars
257
Forks
87
Avg merge
1d 3h
Merged PRs (30d)
29

Description

Version Checks (indicate both or one)
  • I have confirmed this bug exists on the lastest release of Linopy.

  • I have confirmed this bug exists on the current master branch of Linopy.

Issue Description

When a constraint's LHS contains a term whose coefficient is exactly zero, linopy culls that term from the exported matrix, so the constraint's sparsity is data-dependent even though its feasible set is not.

Concretely, for the (common saturation-style) constraint

activation <= ratio * reservation      ratio in [0, 1]

when ratio == 0 the term 0 * reservation is dropped, so the reservation column disappears from that row. Rebuilding the same model with ratio flipped between 0 and a non-zero value produces different LHS sparsity for the very same constraint.

The persistent / warm-start update machinery then cannot update the model in place: persistent/diff.py (diff_con, lines ~454-457) treats a change in the CSR indptr/indices as RebuildReason.SPARSITY, and Solver falls back to a full model rebuild + fresh solve (_rebuild). This defeats warm starts for any dynamic coefficient that crosses zero, which is a very common pattern (e.g. zero-price hours setting a bound to zero).

Root cause (likely introduced in #816) is the zero-coefficient culling at matrix export:

  • Constraint._matrix_export_data (linopy/constraints.py, valid_final = (vars_final != -1) & (coeffs_final != 0))
  • Constraint.flat / to_polars mask_func (mask = (data["vars"] != -1) & (data["coeffs"] != 0))
Reproducible Example
import numpy as np
import linopy
from linopy.persistent.diff import ModelDiff, RebuildReason
from linopy.persistent.snapshot import ModelSnapshot
from linopy.solvers import Solver


def build_model(ratios):
    """Two-timestep model: activation <= ratio * reservation."""
    m = linopy.Model()
    activation = m.add_variables(lower=0, name="activation", coords=[np.arange(2)])
    reservation = m.add_variables(lower=0, name="reservation", coords=[np.arange(2)])
    con = m.add_constraints(activation <= ratios * reservation, name="activation_limit")
    m.add_objective(0 * activation)  # feasibility problem
    return m, con


def row_nnz(m, con):
    csr, _, _, _ = con.to_matrix_with_rhs(m.variables.label_index)
    return np.diff(csr.indptr).tolist()


# (1) Sparsity depends on the data: the zero coefficient is culled.
m1, c1 = build_model(np.array([1.0, 1.0]))   # both rows: activation & reservation
m2, c2 = build_model(np.array([1.0, 0.0]))   # row 2 coefficient == 0
print("ratio=[1, 1]  nnz/row =", row_nnz(m1, c1))   # [2, 2]
print("ratio=[1, 0]  nnz/row =", row_nnz(m2, c2))   # [2, 1]  <- reservation col gone

# (2) The persistent update machinery sees a sparsity change and rebuilds.
snap = ModelSnapshot.capture(m1)
diff = ModelDiff.from_snapshot(snap, m2, same_model=False, ignore_dims=[])
assert diff is RebuildReason.SPARSITY

solver = Solver.from_name("highs", model=m1, io_api="direct", track_updates=True, set_names=False)
solver.solve(assign=True)               # cold solve ok
solver.update(m2, ignore_dims=[])       # warm: only a coefficient changed to 0
assert solver._last_rebuild_reason is RebuildReason.SPARSITY
assert solver._rebuilds == 1            # full rebuild instead of in-place update
print("persistent solver rebuilt (RebuildReason.SPARSITY) instead of updating in place")

Output (verified on v0.9.1; the three assert lines pass, so rebuild_reason == SPARSITY and rebuilds == 1):

ratio=[1, 1]  nnz/row = [2, 2]
ratio=[1, 0]  nnz/row = [2, 1]
persistent solver rebuilt (RebuildReason.SPARSITY) instead of updating in place
Expected Behavior

Linopy should be smarter about the culling, or offer a way to opt-out of it where required

Installed Versions
- linopy 0.9.1 (latest release) - numpy 1.26.x, xarray, scipy - HiGHS 1.15.1 (via linopy)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Constraint._matrix_export_data and the Constraint.flat/to_polars mask_func in linopy/constraints.py, then trace persistent/diff.py around lines 454-457 and Solver._rebuild. Run the reproducible example against the v0.9.1 behavior. Done means zero-valued coefficients no longer cause an unintended sparsity rebuild, or the supported opt-out behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.