deepcopy forgets manual changes to catalog dataframe
- Dominant language
- Python
- Stars
- 164
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
### Description
Using `aggregate=False` in `esm_datastore.to_dataset_dict()` triggers a [deepcopy](https://github.com/intake/intake-esm/blob/bc54ed5dca9088be48bb7801e21aedb39c5d342e/intake_esm/core.py#L583) of the object. For whatever reason, the deepcopy forgets any manual changes made to the dataframe by updating `cat.esmcat._df` (e.g. [as recommended in the documentation here](https://intake-esm.readthedocs.io/en/latest/how-to/manipulate-catalog.html#step-3-attach-the-new-dataframe-to-our-catalog-object)). I would expect manual changes made to the dataframe to be cascaded through the rest of the object.
### What I Did
Replicate the tutorial here: https://intake-esm.readthedocs.io/en/latest/how-to/manipulate-catalog.html
Only change made was to add `aggregate=False` in the call to`cat_subset.to_dataset_dict()`.
Now all 40 original assets are loaded instead of just the 8 intended assets after `cat_subset.esmcat._df` was modified.
### Version information: output of `intake_esm.show_versions()`
Paste the output of `intake_esm.show_versions()` here:
```python
import intake_esm
intake_esm.show_versions()
INSTALLED VERSIONS
------------------
cftime: 1.6.1
dask: 2022.7.1
fastprogress: 1.0.3
fsspec: 2022.7.1
gcsfs: 2022.7.1
intake: 0.6.5
intake_esm: 2021.8.17.post86
netCDF4: 1.6.0
pandas: 1.4.3
requests: 2.28.1
s3fs: 2022.7.1
xarray: 2022.6.0
zarr: 2.12.0
```
Contributor guide
Research direction
Start in intake_esm/core.py around the deepcopy at line 583 and reproduce the catalog-manipulation tutorial with aggregate=False in cat_subset.to_dataset_dict(). Verify that manual changes to cat_subset.esmcat._df are preserved and that the result loads the intended 8 assets rather than all 40.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100