pydata / pydata/xarray

open_mfdataset for one file, time subsetting

Open
#4,626 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

Hi,

Until now, we used open_mfdataset to open 12 datasets (one corresponding to a month). It works perfectly. But now I have those data in one dataset covering 26 years. So I used sel to subset the dataset and get the 12 wanted months.

 with xr.open_mfdataset(file_list, combine="by_coords", parallel=True) as ds: 
    ld = pd.date_range(start="20170101", end="20171201", freq="MS") 
    ds = ds.sel({"time":ld}, method="nearest")

With file_list is a list of just one file.

When I print ds, I correctly get a dataset with the size 12 for the time (dask.array<chunksize=(12, 75, 681, 1440))

But later in the code, I get "MemoryError: Unable to allocate 85.5 GiB for an array with shape (312, 75, 681, 1440) and data type float32".
In this error you can see that the time variable got back to 312 (26 years).

But oddly, when I change open_mfdataset(file_list,..) by open_dataset(file_list[0]), it works. I still have the shape (12, 75, 681, 1440).

Do you have any idea for this behavior ?

Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the comparison between xr.open_mfdataset(file_list, combine="by_coords", parallel=True) and xr.open_dataset(file_list[0]) using the one-file, 26-year dataset and the 2017 monthly selection. Trace why the later operation sees 312 time steps; done means explaining or correcting the differing behavior without the 85.5 GiB allocation while retaining the 12 selected times.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.