intake / intake/intake-esm

deepcopy forgets manual changes to catalog dataframe

Open
#496 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

### Description

Using `aggregate=False` in `esm_datastore.to_dataset_dict()` triggers a [deepcopy](https://github.com/intake/intake-esm/blob/bc54ed5dca9088be48bb7801e21aedb39c5d342e/intake_esm/core.py#L583) of the object. For whatever reason, the deepcopy forgets any manual changes made to the dataframe by updating `cat.esmcat._df` (e.g. [as recommended in the documentation here](https://intake-esm.readthedocs.io/en/latest/how-to/manipulate-catalog.html#step-3-attach-the-new-dataframe-to-our-catalog-object)). I would expect manual changes made to the dataframe to be cascaded through the rest of the object.

### What I Did

Replicate the tutorial here: https://intake-esm.readthedocs.io/en/latest/how-to/manipulate-catalog.html

Only change made was to add `aggregate=False` in the call to`cat_subset.to_dataset_dict()`.

Now all 40 original assets are loaded instead of just the 8 intended assets after `cat_subset.esmcat._df` was modified.

### Version information: output of `intake_esm.show_versions()`

Paste the output of `intake_esm.show_versions()` here:

```python
import intake_esm

intake_esm.show_versions()

INSTALLED VERSIONS
------------------

cftime: 1.6.1
dask: 2022.7.1
fastprogress: 1.0.3
fsspec: 2022.7.1
gcsfs: 2022.7.1
intake: 0.6.5
intake_esm: 2021.8.17.post86
netCDF4: 1.6.0
pandas: 1.4.3
requests: 2.28.1
s3fs: 2022.7.1
xarray: 2022.6.0
zarr: 2.12.0
```

Contributor guide

Open the contributing guide

Research direction

Start in intake_esm/core.py around the deepcopy at line 583 and reproduce the catalog-manipulation tutorial with aggregate=False in cat_subset.to_dataset_dict(). Verify that manual changes to cat_subset.esmcat._df are preserved and that the result loads the intended 8 assets rather than all 40.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.