Downsampling with resample generates longer time array than expected
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
MCVE Code Sample
import xarray as xr
import pandas as pd
from functools import reduce
# Create time dimension array for all climatological winters
indexes = [pd.DatetimeIndex(f'{year}-12-01', f'{year+1}-03-01', freq='12H')
for year in range(1980, 2020)]
index_union = reduce(pd.Index.union, indexes)
# Create DataArray
ds = xr.Dataset({'var': ('time', np.arange(len(ix_union))), 'time': index_union})
Problem Description
From here we can check ds time dimension months:
>>> pd.DatetimeIndex(ds.time.values).month.unique()
Int64Index([12, 1, 2, 3], dtype='int64')
Now, if we downsample our Dataset to a one week period:
>>> pd.DatetimeIndex(ds.resample(time='1W').mean().time.values).month.unique()
Int64Index([12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], dtype='int64')
Instead of getting the weeks within the months in the original Dataset, we obtain additional months with missing values in the var array. One way of solving this issue is to use the dropna() method, but is a slow approach.
Expected Output
We would expect something like this:
>>> pd.DatetimeIndex(ds.resample(time='1W').mean().time.values).month.unique()
Int64Index([12, 1, 2, 3], dtype='int64')
Output of xr.show_versions()
xarray: 0.14.1
pandas: 0.25.3
numpy: 1.17.5
scipy: 1.4.1
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.0.4.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: 0.9.7.7
iris: 2.3.0
bottleneck: None
dask: 2.9.2
distributed: 2.9.3
matplotlib: 3.1.1
cartopy: 0.17.0
seaborn: 0.9.0
numbagg: None
setuptools: 44.0.0.post20200106
pip: 19.3.1
conda: None
pytest: None
IPython: 7.11.1
sphinx: None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the MCVE around Dataset.resample(time='1W').mean(), correcting the apparent np and ix_union typos if needed. Trace the resampling entry point and compare the generated time bins with the non-missing var values. Done means resampling the seasonal dataset does not create empty bins for unrelated months, without requiring dropna().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100