intake / intake/intake-esm

Proper way to handle failing `preprocess` output.

Open
#332 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
164
Forks
54
PR merge metrics
No merged PRs in 30d

Description

I am encountering an issue with one dataset when loading many CMIP6 datasets using intake-esm (see #331).

I believe this is actually an issue with the raw data, but either way it got me curious if there is a way to handle the following scenario properly:

Lets say I have 2 dataset (`ds_a`,`ds_b`) in 2 different zarr stores and an appropriately set up intake-esm catalog.
Now I have some preprocessing function `func`.

`func` modifies something on each datasets, works fine on `ds_a`, but fails on `ds_b`.
Currently that will lead to a complete failure when reading in the full catalog with `.to_datasets_dict()`.

Is there a way to simply exclude the failing dataset but continue to process only the ones that work? This would be very helpful to me.

EDIT: In further investigating this, it seems that in #331 the preprocessing is not even needed, but I guess this question can be phrased more generally: Is there a way to still output some datasets if errors are coming up for some of them?

Contributor guide

Open the contributing guide

Research direction

Start at the catalog-loading entry point used by `.to_datasets_dict()` and trace how preprocessing errors from individual datasets are propagated. Define how failing datasets should be excluded while successful datasets remain available, including the expected error reporting, then add coverage for one successful and one failing dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.