pydata / pydata/xarray

An `asfreq` method without `resample`, and clarify or improve resample().asfreq() behavior for down-sampling

Open
#3,242 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

MCVE Code Sample
# Your code here
>>> import numpy as np
>>> import xarray as xr
>>> import pandas as pd

>>> data = np.random.random(300)

# Make a time grid that doesn't start exactly on the hour.
>>> time = pd.date_range('2019-01-01', periods=300, freq='T') + pd.Timedelta('3T')
>>> time
DatetimeIndex(['2019-01-01 00:03:00', '2019-01-01 00:04:00',
               '2019-01-01 00:05:00', '2019-01-01 00:06:00',
               '2019-01-01 00:07:00', '2019-01-01 00:08:00',
               '2019-01-01 00:09:00', '2019-01-01 00:10:00',
               '2019-01-01 00:11:00', '2019-01-01 00:12:00',
               ...
               '2019-01-01 04:53:00', '2019-01-01 04:54:00',
               '2019-01-01 04:55:00', '2019-01-01 04:56:00',
               '2019-01-01 04:57:00', '2019-01-01 04:58:00',
               '2019-01-01 04:59:00', '2019-01-01 05:00:00',
               '2019-01-01 05:01:00', '2019-01-01 05:02:00'],
              dtype='datetime64[ns]', length=300, freq='T')

>>> da = xr.DataArray(data, dims=['time'], coords={'time': time})
>>> resampled = da.resample(time='H').asfreq()
>>> resampled
<xarray.DataArray (time: 6)>
array([0.478601, 0.488425, 0.496322, 0.479256, 0.523395, 0.201718])
Coordinates:
  * time     (time) datetime64[ns] 2019-01-01 ... 2019-01-01T05:00:00

# The value is actually the mean over the time window, eg. the third value is:
>>> da.loc['2019-01-01T02:00:00':'2019-01-01T02:59:00'].mean()
<xarray.DataArray ()>
array(0.496322)
Expected Output

Docs say this:

Return values of original object at the new up-sampling frequency; 
essentially a re-index with new times set to NaN.

I suppose this doc is not technically wrong, since upon careful reading, I realize it does not define a behavior for down-sampling. But it's easy to: (1) assume the same behavior (reindexing) for down-sampling and up-sampling and/or (2) expect behavior similar to df.asfreq() in pandas.

Problem Description

I would argue for an asfreq method without resampling that matches the pandas behavior, which AFAIK, is to reindex starting at the first timestamp, at the specified interval.

>>> df = pd.DataFrame(da, index=time)
>>> df.asfreq('H')
                            0
2019-01-01 00:03:00  0.065304
2019-01-01 01:03:00  0.325814
2019-01-01 02:03:00  0.841201
2019-01-01 03:03:00  0.610266
2019-01-01 04:03:00  0.613906

This can currently easily be achieved, so it's not a blocker.

>>> da.reindex(time=pd.date_range(da.time[0].values, da.time[-1].values, freq='H'))
<xarray.DataArray (time: 5)>
array([0.065304, 0.325814, 0.841201, 0.610266, 0.613906])
Coordinates:
  * time     (time) datetime64[ns] 2019-01-01T00:03:00 ... 2019-01-01T04:03:00

Why I argue for asfreq functionality outside of resampling is that asfreq(freq) in pandas is purely a reindex, compared to eg resample(freq).first() which would give you a different time index.

Output of xr.show_versions()

Still on python27, show_versions actually throws an exception, because some HDF5 library doesn't have a magic property. I don't think this detail is relevant here though.

``` >>> xr.__version__ u'0.11.3' ```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the resample(time='H').asfreq() behavior shown in the MCVE and compare it with pandas DataFrame.asfreq() and the suggested reindex approach. Define whether a standalone asfreq method, clarified down-sampling semantics, or both are needed; done means the chosen behavior is documented and supported by matching examples or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.