xr.DataSet.from_dataframe / xr.DataArray.from_series does not preserve DateTimeIndex with timezone
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
Problem Description
When using DataSet.from_dataframe (DataArray.from_series) to convert a pandas dataframe with DateTimeIndex having a timezone - xarray convert the datetime into a nanosecond index - rather than keeping it as a datetime-index type.
MCVE Code Sample
print( df.index )
DatetimeIndex(['2000-01-03 16:00:00-05:00', '2000-01-03 16:00:00-05:00',
'2000-01-03 16:00:00-05:00', '2000-01-03 16:00:00-05:00',
...
'2019-08-20 16:00:00-05:00', '2019-08-20 16:00:00-05:00'],
dtype='datetime64[ns, EST]', name='time', length=12713014, freq=None)
ds = xr.DataSet.from_dataframe( df.head( 1000 ) )
print( ds['time'] )
<xarray.DataArray 'time' (time: 7)>
array([946933200000000000, 947019600000000000, 947106000000000000,
947192400000000000, 947278800000000000, 947538000000000000,
947624400000000000, ...], dtype=object)
Coordinates:
* time (time) object 946933200000000000 ... 947624400000000000
Expected Output
After removing the tz localization from the DateTimeIndex of the dataframe , the conversion to a DataSet preserves the time-index (without converting it to nanoseconds)
df.index = df.index.tz_convert('UTC').tz_localize(None)
ds = xr.DataSet.from_dataframe( df.head(1000) )
print( ds['time] )
<xarray.DataArray 'time' (time: 7)>
array(['2000-01-03T21:00:00.000000000', '2000-01-04T21:00:00.000000000',
'2000-01-05T21:00:00.000000000', '2000-01-06T21:00:00.000000000',
'2000-01-07T21:00:00.000000000', '2000-01-10T21:00:00.000000000',
'2000-01-11T21:00:00.000000000'], dtype='datetime64[ns]')
Coordinates:
* time (time) datetime64[ns] 2000-01-03T21:00:00 ... 2000-01-11T21:00:00
Output of xr.show_versions()
xarray: 0.12.3+81.g41fecd86
pandas: 0.24.2
numpy: 1.16.2
scipy: 1.2.1
netCDF4: None
pydap: None
h5netcdf: None
h5py: 2.9.0
Nio: None
zarr: None
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: 1.1.4
distributed: 1.26.0
matplotlib: 3.0.3
cartopy: None
seaborn: 0.9.0
numbagg: None
setuptools: 40.8.0
pip: 19.0.3
conda: 4.7.11
pytest: 4.3.1
IPython: 7.4.0
sphinx: 1.8.5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the MCVE against DataSet.from_dataframe and DataArray.from_series with the timezone-aware pandas DatetimeIndex shown in the issue. Trace those conversion entry points and verify that the resulting time coordinate preserves datetime values and timezone information rather than becoming nanosecond integers; compare it with the timezone-localized example as the expected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100