Proper way to handle failing `preprocess` output.
- Dominant language
- Python
- Stars
- 164
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
I am encountering an issue with one dataset when loading many CMIP6 datasets using intake-esm (see #331).
I believe this is actually an issue with the raw data, but either way it got me curious if there is a way to handle the following scenario properly:
Lets say I have 2 dataset (`ds_a`,`ds_b`) in 2 different zarr stores and an appropriately set up intake-esm catalog.
Now I have some preprocessing function `func`.
`func` modifies something on each datasets, works fine on `ds_a`, but fails on `ds_b`.
Currently that will lead to a complete failure when reading in the full catalog with `.to_datasets_dict()`.
Is there a way to simply exclude the failing dataset but continue to process only the ones that work? This would be very helpful to me.
EDIT: In further investigating this, it seems that in #331 the preprocessing is not even needed, but I guess this question can be phrased more generally: Is there a way to still output some datasets if errors are coming up for some of them?
Contributor guide
Research direction
Start at the catalog-loading entry point used by `.to_datasets_dict()` and trace how preprocessing errors from individual datasets are propagated. Define how failing datasets should be excluded while successful datasets remain available, including the expected error reporting, then add coverage for one successful and one failing dataset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100