open_mfdataset for one file, time subsetting
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Hi,
Until now, we used open_mfdataset to open 12 datasets (one corresponding to a month). It works perfectly. But now I have those data in one dataset covering 26 years. So I used sel to subset the dataset and get the 12 wanted months.
with xr.open_mfdataset(file_list, combine="by_coords", parallel=True) as ds:
ld = pd.date_range(start="20170101", end="20171201", freq="MS")
ds = ds.sel({"time":ld}, method="nearest")
With file_list is a list of just one file.
When I print ds, I correctly get a dataset with the size 12 for the time (dask.array<chunksize=(12, 75, 681, 1440))
But later in the code, I get "MemoryError: Unable to allocate 85.5 GiB for an array with shape (312, 75, 681, 1440) and data type float32".
In this error you can see that the time variable got back to 312 (26 years).
But oddly, when I change open_mfdataset(file_list,..) by open_dataset(file_list[0]), it works. I still have the shape (12, 75, 681, 1440).
Do you have any idea for this behavior ?
Thank you.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the comparison between xr.open_mfdataset(file_list, combine="by_coords", parallel=True) and xr.open_dataset(file_list[0]) using the one-file, 26-year dataset and the 2017 monthly selection. Trace why the later operation sees 312 time steps; done means explaining or correcting the differing behavior without the 85.5 GiB allocation while retaining the 12 selected times.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100