Deltares / Deltares/imod-python
open_projectfile_data may allocate big arrays eagerly for constants in the project file?
@HendrikKok is already working on this.
Since Feb 14, 2024.
- Dominant language
- Python
- Stars
- 41
- Forks
- 12
- Avg merge
- 21h 8m
- Merged PRs (30d)
- 1
Description
I haven't checked this very carefully, but there are breadcrumbs which may be useful...
In a project file, it's possible to mix IDF and constant data willy-nilly for variables. We cannot directly represent such data in xarray because if e.g. a recharge rate has x and y coordinates for the first timestamp, we cannot join it with a single constant value for the second timestamp.
For this reason, _create_dataarrays:
https://github.com/Deltares/imod-python/blob/b59a7700a75d10bf33ed51c6e532bf89ac625410/imod/formats/prj/prj.py#L530
calls two methods internally, _from_paths (IDFs) and _from_values (constants values provided inline). The idea is constants are expanded to merge with potential IDF data. That's this function:
def _create_dataarray_from_values(values: List[float], headers: List[Dict[str, Any]]) -> xr.DataArray:
coords = _merge_coords(headers)
firstdims = headers[0]["dims"]
shape = [len(coord) for coord in coords.values()]
da = xr.DataArray(np.reshape(values, shape), dims=firstdims, coords=coords)
return da
There might be a problem with the final line. Since it's calling numpy reshape, it will allocate a numpy array eagerly.
We should probably create a dask-based DataArray instead, chunking in time if there's a time coordinate.
I've also just noticed that the mixed case might not work at all...
In the test, edit this line:
into: 1,2, 001, 1.0, 0.0, 1.0,
And run the test.
@HendrikKok: to see if you can replicate @jc-hunink's problem, you can try specifying 700 constant values for the recharge and check whether you can succesfully run open_projectfile_data.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.