Deltares / Deltares/imod-python

`Modflow6Simulation.from_file()` or even Package instantiation could always eagerly materialize all non-spatial data

Open
#1,689 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
41
Forks
12
Avg merge
21h 8m
Merged PRs (30d)
1

Description

This has some relation to #1686

My finding there is that .item() doesn't work on dask-backed scalar values. This looks like an issue/inconsistency with the xarray zarr backend (in comparison with netCDF). The workaround is to simply force all non-spatial variables into memory.

However, this workaround might have general benefits: forcing a compute on these small (non-spatial) variables will decrease IO overhead in general, such that it may be worthwhile to always call a .load() or .compute(). Alternatively, it has no downsides (apart from the single loop over the dataset variables at instantiation).

An argument could be made that e.g. the contents of the DIS package should always be loaded into RAM as well.

We could even take this further: guarantee that some arrays are materialized (i.e. numpy backed instead of dask backed) at package instantiation. The resulting memory pressure is small, but it might increase performance predictability.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Modflow6Simulation.from_file() and package instantiation, then reproduce the dask-backed scalar .item() failure related to #1686. Review how non-spatial variables and DIS package contents are currently loaded, and compare the xarray zarr behavior with netCDF. Done requires a clearly chosen materialization scope and validation of the resulting behavior and performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.