intake / intake/intake-esm

`require_all_on` fails with a derived variable registry

Open
#436 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

Here's a quick checklist in what to include:

- [x] Include a detailed description of the bug or suggestion
- [x] Output of `intake_esm.show_versions()`
- [x] Minimal, self-contained copy-pastable example that generates the issue if possible. Please be concise with code posted.

### Description
When searching a catalog that has a derived variable registry and specifying `require_all_on`, the output is incorrect.

csv file:
```csv
simulation_id,ensemble_id,model_institution_id,model_id,experiment_id,timestep_id,domain_id,member_id,variable_id,file
A,CMIP6,CCCma,CanESM,historical,day,NAM,r1i1p1,tasmax,file1
A,CMIP6,CCCma,CanESM,historical,day,NAM,r1i1p1,pr,file2
B,CMIP6,CCCmb,CanESM,historical,day,NAM,r1i1p1,pr,file3
C,CMIP6,CCCmc,CanESM,historical,day,NAM,r1i1p1,tasmax,file4
D,CMIP6,CCCmd,CanESM,historical,day,NAM,r1i1p1,pr,file5
D,CMIP6,CCCmd,CanESM,historical,day,NAM,r1i1p1,prsn,file6
D,CMIP6,CCCmd,CanESM,historical,day,NAM,r1i1p1,tasmax,file7
```
json file:
```
{
"esmcat_version": "0.1.0",
"assets": {
"column_name": "file",
"format": "netCDF"
},
"aggregation_control": {
"variable_column_name": "variable_id",
"groupby_attrs": ["simulation_id", "domain_id", "timestep_id"],
"aggregations": [
{"type": "join_new", "attribute_name": "member_id"},
{"type": "union", "attribute_name": "variable_id"}
],
},
"attributes" : [],
"catalog_file": "test.csv"
}
```

### What I Did
```python3
import ast
import intake
import intake_esm

dvr = intake_esm.DerivedVariableRegistry()

@dvr.register(variable='prsn', query={'variable_id': 'pr'})
def prsn(ds):
return ds.pr

cat = intake.open_esm_datastore('test.json', registry=dvr)
cat.search(variable_id=['tasmax', 'prsn'], require_all_on=['simulation_id']).df
```
gives
```
simulation_id ensemble_id model_institution_id model_id experiment_id timestep_id domain_id member_id variable_id file
0 D CMIP6 CCCmd CanESM historical day NAM r1i1p1 prsn file6
1 D CMIP6 CCCmd CanESM historical day NAM r1i1p1 tasmax file6
2 A CMIP6 CCCma CanESM historical day NAM r1i1p1 pr file2
3 B CMIP6 CCCmb CanESM historical day NAM r1i1p1 pr file3
4 D CMIP6 CCCmd CanESM historical day NAM r1i1p1 pr file5
```
- Line 4 is unneeded, but I can live with this.
- Line 3 is erroneous because `B` does not have "tasmax", so it shouldn't have been in the output.
- `A` doesn't include the `tasmax` variable. It does exist, so I expected it to be there.

I don't know how to solve this one though...

### Version information: output of `intake_esm.show_versions()`

INSTALLED VERSIONS
------------------

cftime: 1.5.1.1
dask: 2021.12.0
fastprogress: 0.2.7
fsspec: 2021.11.1
gcsfs: 2021.11.1
intake: 0.6.4
intake_esm: 2021.8.17.post43+dirty
netCDF4: 1.5.8
pandas: 1.3.5
requests: 2.26.0
s3fs: 2021.11.1
xarray: 0.20.2
zarr: 2.10.3

Contributor guide

Open the contributing guide

Research direction

Start by running the provided Python example with intake.open_esm_datastore and cat.search(..., require_all_on=['simulation_id']). Then inspect DerivedVariableRegistry handling and the require_all_on filtering path; done when simulations are included only if the requested variables are actually present and the returned dataframe has the expected assets.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.