GenericMappingTools / GenericMappingTools/pygmt

xarray.Dataset should NOT be recognized as a valid tabular data type

Open
#3,146 6 comments 1 reaction 0 assignees View on GitHub
discussions
Dominant language
Python
Stars
874
Forks
255
Avg merge
1d 21h
Merged PRs (30d)
40

Description

Currently, [`xarray.Dataset`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html) is recognized as a valid tabular data type in some places. For example:

https://github.com/GenericMappingTools/pygmt/blob/dbbc1684ded29ec8d8be7db727abdffd62538d2b/pygmt/helpers/utils.py#L108-L109
https://github.com/GenericMappingTools/pygmt/blob/dbbc1684ded29ec8d8be7db727abdffd62538d2b/pygmt/clib/session.py#L1531-L1547

But I think it should **NOT** be like that. Here are the reasons.

**1. `xarray.Dataset` is more like a collection of xarray.DataArrays, rather than a pandas.DataFrame:**

As the [official docs](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html) says:

> A dataset resembles an in-memory representation of a NetCDF file, and consists of variables, coordinates and attributes which together form a self describing dataset.
>
> Dataset implements the mapping interface with keys given by variable names and values given by DataArray objects for each variable name.

`xarray.Dataset` can represent tabular data, but it's more commonly used as a data structure to hold multiple `xarray.DataArray` objects.

**2. It's unclear what/how data are passed.**

Here is an example from the official documentation:
```
>>> import numpy as np
>>> import xarray as xr
>>> import pandas as pd
>>> np.random.seed(0)
>>> temperature = 15 + 8 * np.random.randn(2, 2, 3)
>>> precipitation = 10 * np.random.rand(2, 2, 3)
>>> lon = [[-99.83, -99.32], [-99.79, -99.23]]
>>> lat = [[42.25, 42.21], [42.63, 42.59]]
>>> time = pd.date_range("2014-09-06", periods=3)
>>> reference_time = pd.Timestamp("2014-09-05")

>>> ds = xr.Dataset(
... data_vars=dict(
... temperature=(["x", "y", "time"], temperature),
... precipitation=(["x", "y", "time"], precipitation),
... ),
... coords=dict(
... lon=(["x", "y"], lon),
... lat=(["x", "y"], lat),
... time=time,
... reference_time=reference_time,
... ),
... attrs=dict(description="Weather related data."),
... )
>>> ds
Size: 288B
Dimensions: (x: 2, y: 2, time: 3)
Coordinates:
lon (x, y) float64 32B -99.83 -99.32 -99.79 -99.23
lat (x, y) float64 32B 42.25 42.21 42.63 42.59
* time (time) datetime64[ns] 24B 2014-09-06 2014-09-07 2014-09-08
reference_time datetime64[ns] 8B 2014-09-05
Dimensions without coordinates: x, y
Data variables:
temperature (x, y, time) float64 96B 29.11 18.2 22.83 ... 16.15 26.63
precipitation (x, y, time) float64 96B 5.68 9.256 0.7104 ... 4.615 7.805
Attributes:
description: Weather related data.
```
Each data variable is a `xarray.DataArray` object:
```
>>> ds.temperature
Size: 96B
array([[[29.11241877, 18.20125767, 22.82990387],
[32.92714559, 29.94046392, 7.18177696]],

[[22.60070734, 13.78914233, 14.17424919],
[18.28478802, 16.15234857, 26.63418806]]])
Coordinates:
lon (x, y) float64 32B -99.83 -99.32 -99.79 -99.23
lat (x, y) float64 32B 42.25 42.21 42.63 42.59
* time (time) datetime64[ns] 24B 2014-09-06 2014-09-07 2014-09-08
reference_time datetime64[ns] 8B 2014-09-05
Dimensions without coordinates: x, y
```
Then, in [`virtualfile_in`](https://github.com/GenericMappingTools/pygmt/blob/dbbc1684ded29ec8d8be7db727abdffd62538d2b/pygmt/clib/session.py#L1597), a list of multi-dimensional (3-D in this example) `xarray.DataArray` objects are passed to GMT modules which expects a list of 1-D arrays instead. It works without errors because in `Session.put_vector`, we pass the pointer of the 2-D array to the GMT C API function, but it likely won't work if the data is not C-contiguous (e.g., a slice of a dataset). So, the actual behavior is not well defined.

So, I think `xarray.Dataset` should not be recognized as a valid tabular data type, which not only makes more sense but also can simplify our codes/tests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.