intake / intake/intake-esm

Drop non-requested variables from datasets

Open
#448 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
When using the catalog with a derived variable registry and calling `to_dataset_dict`, all variables that were needed for the derived variable computation are in the final datasets, even if they were not requested directly.

**Describe the solution you'd like**
In the following imaginary example, say we have datasets for `tasmin`, `tasmax` and `pr` and a function for `tas = func(tasmin, tasmax)` in the DVR.
```python3
subcat = cat.search(variable=['tas', 'pr'])
subcat.to_dataset_dict()
```
I would expect the ouput dataset to _only_ include `tas` and `pr`, but with the current intake-esm `tasmin` and `tasmax` are also returned.

**Describe alternatives you've considered**
1. Wrapping it myself. Remembering what I wanted in a list and dropping the rest.
2. Instead of passing `_requested_variables` to the `ESMDataSource` objects, perform that subset in `to_dataset_dict` directly, after the derived variables have been created.
3. Add an aditionnal `_needed_variables` that would store a list of variables including dependencies, while `_requested_variables` would not list dependencies. And a subset would be performed by `to_dataset_dict` after the derived variable creation.

Contributor guide

Open the contributing guide

Research direction

Start by tracing to_dataset_dict and how _requested_variables is passed to ESMDataSource objects, then inspect the derived variable registry path. Done means a request for derived and ordinary variables returns only those requested variables while retaining the derived result; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.