Deltares / Deltares/imod-python
`Modflow6Simulation.from_file()` or even Package instantiation could always eagerly materialize all non-spatial data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 41
- Forks
- 12
- Avg merge
- 21h 8m
- Merged PRs (30d)
- 1
Description
This has some relation to #1686
My finding there is that .item() doesn't work on dask-backed scalar values. This looks like an issue/inconsistency with the xarray zarr backend (in comparison with netCDF). The workaround is to simply force all non-spatial variables into memory.
However, this workaround might have general benefits: forcing a compute on these small (non-spatial) variables will decrease IO overhead in general, such that it may be worthwhile to always call a .load() or .compute(). Alternatively, it has no downsides (apart from the single loop over the dataset variables at instantiation).
An argument could be made that e.g. the contents of the DIS package should always be loaded into RAM as well.
We could even take this further: guarantee that some arrays are materialized (i.e. numpy backed instead of dask backed) at package instantiation. The resulting memory pressure is small, but it might increase performance predictability.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at Modflow6Simulation.from_file() and package instantiation, then reproduce the dask-backed scalar .item() failure related to #1686. Review how non-spatial variables and DIS package contents are currently loaded, and compare the xarray zarr behavior with netCDF. Done requires a clearly chosen materialization scope and validation of the resulting behavior and performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100