PyPSA / PyPSA/linopy

Support for sparse xarrays

Open
#248 34 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

data-model enhancement sparse
Dominant language
Python
Stars
257
Forks
87
Avg merge
1d 3h
Merged PRs (30d)
29

Description

First of, great work with the library so far!! This is significantly better than what currently exists out there.

Many energy systems models can be extremely sparse. For example, in one recent case, a linear expression is composed of the sum of only 0.1% of the coordinates of an xarray. Not supporting sparsity hence would increase our model size by ~900x. This effectively prohibits us from using Linopy as is.

Suggested solution

Ideally linopy could support xarrays based on the sparse library. I've tried the library but it doesn't work with linopy particularly when trying to combine sparse and not sparse xarrays.

Rejected solutions

  1. I tried using MultiIndices which are natively supported in xarray however broadcasting doesn't work when combining a sub-dimension of a multi-index. In general, my gut tells me this is the wrong direction to take for the library.

  2. I'm aware of the mask parameter for creating constraints and variables however this a) still requires a large memory allocation and b) doesn't help during the computation of linear expressions (it only helps at the variable/constraint creation stage).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points. Start by reproducing the reported failure when combining sparse and non-sparse xarrays in linopy, then trace the relevant xarray operations. Done means sparse-library xarrays are supported without the large dense allocation described.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.