Deltares / Deltares/imod-python

`open_projectfile_data`: rechunk to have chunks only in time, or rechunk in conversion

Open
#1,144 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Python
Stars
41
Forks
12
Avg merge
21h 8m
Merged PRs (30d)
1

Description

Reconsidering #845 had me thinking: we're creating a very large number of tasks if the chunks are sized 1 along both time and layer.

This is a direct consequence of the IDF reading which creates one task for each file. However, the conversion to MF6 will generally operate on all the layers at once. We can greatly reduce the number of tasks and the size of the task graph by making sure chunks are merged in the layer dimension.

I'm only doubtful whether we should do this directly in open_projectfile_data such that any user would benefit from this different default. There is a downside, let's say you do something like this this:

prj_data = imod.prj.open_projectfile_data(stuff)

khv = prj_data["khv"]
for layer in khv["layer"]:
     khv.sel(layer=layer).plot()

This will now load all IDFs uselessly for plotting a single layer. Of course, a simple .compute() addresses it:

khv = prj_data["khv"].compute()

But I do not expect most users to come up with this themselves.

Many other operations would probably work better without layer chunking though. Of course, the same is true for layer chunking in imod.idf.open and I haven't seen any complaints from that.

So what I'd suggest now is to remove the layer chunks in the from_imod5_data method (i.e. set layer chunk size equal to dimension size):

chunksizes = dict(da.chunksizes)
if "layer" in chunksizes:
     chunksizes["layer"] = (da.sizes["layer"],)
     da = da.chunk(chunksizes)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the from_imod5_data method and tracing how chunksizes are derived from the converted data. Check the resulting chunk layout for layer and time dimensions, including the single-layer plotting case. Done means layer chunks are merged as proposed without breaking the conversion behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.