Deltares / Deltares/imod-python

open_projectfile_data may allocate big arrays eagerly for constants in the project file?

Open
#845 2 comments 0 reactions 1 assignee View on GitHub

@HendrikKok is already working on this.

Since Feb 14, 2024.

Dominant language
Python
Stars
41
Forks
12
Avg merge
21h 8m
Merged PRs (30d)
1

Description

I haven't checked this very carefully, but there are breadcrumbs which may be useful...

In a project file, it's possible to mix IDF and constant data willy-nilly for variables. We cannot directly represent such data in xarray because if e.g. a recharge rate has x and y coordinates for the first timestamp, we cannot join it with a single constant value for the second timestamp.

For this reason, _create_dataarrays:
https://github.com/Deltares/imod-python/blob/b59a7700a75d10bf33ed51c6e532bf89ac625410/imod/formats/prj/prj.py#L530

calls two methods internally, _from_paths (IDFs) and _from_values (constants values provided inline). The idea is constants are expanded to merge with potential IDF data. That's this function:

def _create_dataarray_from_values(values: List[float], headers: List[Dict[str, Any]]) -> xr.DataArray:
    coords = _merge_coords(headers)
    firstdims = headers[0]["dims"]
    shape = [len(coord) for coord in coords.values()]
    da = xr.DataArray(np.reshape(values, shape), dims=firstdims, coords=coords)
    return da

There might be a problem with the final line. Since it's calling numpy reshape, it will allocate a numpy array eagerly.
We should probably create a dask-based DataArray instead, chunking in time if there's a time coordinate.

I've also just noticed that the mixed case might not work at all...
In the test, edit this line:

https://github.com/Deltares/imod-python/blob/b59a7700a75d10bf33ed51c6e532bf89ac625410/imod/tests/test_formats/test_prj.py#L427

into: 1,2, 001, 1.0, 0.0, 1.0,

And run the test.

@HendrikKok: to see if you can replicate @jc-hunink's problem, you can try specifying 700 constant values for the recharge and check whether you can succesfully run open_projectfile_data.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.