Problematic behavior for combining dataset attributes.
- Dominant language
- Python
- Stars
- 164
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
I have been trying to detrend CMIP6 data using the metadata like `branch_time_in_parent` from models with multiple members and gotten some really confusing results. I was not able to align e.g. the historical members with the control run using that data.
I dug a little deeper and it turns out some models do branch of members at different times. For the examples I looked at, each member had the correct date encoded in their dataset attributes, but upon combining them with intake-esm, errors can occur.
I wrote a 'shortish' example to demonstrate, what I think is a quite undesirable behavior:
First create some dummy datasets with a single attribute that is different
```
# create 3 dummy datasets
import xarray as xr
import numpy as np
for starttime in range(3):
ds = xr.DataArray(np.random.rand(3)).to_dataset(name='data')
ds.attrs['start']=starttime
ds.to_zarr(f'test_{starttime}')
```
Create a matching catalog file
```
# create a corresponding catalog file
import pandas as pd
pd.DataFrame(
{
'zstore':['test_0', 'test_1', 'test_2'],
"member_id":range(3),
'activity_id':['test', 'test', 'test'],
'institution_id':['test', 'test', 'test'],
'source_id':['test', 'test', 'test'],
'experiment_id':['test', 'test', 'test'],
'grid_label':['test', 'test', 'test'],
'table_id':['test', 'test', 'test'],
'variable_id':['data', 'data', 'data'],
}).to_csv('test.csv.gz')
```
As a collection file I am using a slightly modified version (only the catalog path) of the pangeo CMIP6 file
{
"esmcat_version": "0.1.0",
"id": "glade-cmip6",
"description": "This is an ESM collection for CMIP6 data accessible on the Princeton University disk storage system in /scratch/gpfs/GEOCLIM/synda/data/CMIP6",
"catalog_file": "./test.csv.gz",
"attributes": [
{
"column_name": "activity_id",
"vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_activity_id.json"
},
{
"column_name": "source_id",
"vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_source_id.json"
},
{
"column_name": "institution_id",
"vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_institution_id.json"
},
{
"column_name": "experiment_id",
"vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_experiment_id.json"
},
{ "column_name": "member_id", "vocabulary": "" },
{
"column_name": "table_id",
"vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_table_id.json"
},
{ "column_name": "variable_id", "vocabulary": "" },
{
"column_name": "grid_label",
{
"column_name": "table_id",
"vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_table_id.json"
},
{ "column_name": "variable_id", "vocabulary": "" },
{
"column_name": "grid_label",
"vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_grid_label.json"
},
{
"column_name": "version",
"vocabulary": ""
},
{
"column_name": "dcpp_start_year",
"vocabulary": ""
}
],
"assets": {
"column_name": "zstore",
"format": "zarr"
},
"aggregation_control": {
"variable_column_name": "variable_id",
"groupby_attrs": [
"activity_id",
"institution_id",
"source_id",
"experiment_id",
"table_id",
"grid_label"
],
"aggregations": [
{
"type": "union",
"attribute_name": "variable_id"
},
{
"type": "join_new",
"attribute_name": "member_id",
"options": { "coords": "minimal", "compat": "override" }
}
]
}
}
Then I open with intake-esm and convert to a dataset dictionary
```python
import intake
col = intake.open_esm_datastore('test.json')
dd = col.to_dataset_dict()
dd
```
which gives:
```
{'test.test.test.test.test.test':
Dimensions: (dim_0: 3, member_id: 3)
Coordinates:
* member_id (member_id) int64 0 1 2
Dimensions without coordinates: dim_0
Data variables:
data (member_id, dim_0) float64 dask.array
Attributes:
start: 2
intake_esm_varname: data
intake_esm_dataset_key: test.test.test.test.test.test}
```
It seems like the attribute `start` was just overwritten with the value of the last read dataset. I wonder if there is a way to parse this out. Perhaps an option to promote each attribute which differs to a coordinate (with dimension `member_id`), so that this information is preserved in the most detail possible?
### Output of ``intake_esm.__version__``
```
'2020.6.11'
```
Contributor guide
Assessment
This issue has not been assessed yet.