Climate-REF / Climate-REF/climate-ref

finalise_datasets aborts the whole solve when a model stores a numeric attribute as a string

Open
#838 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
26
Forks
14
Avg merge
1d 10h
Merged PRs (30d)
36

Description

`FinaliseableDatasetAdapterMixin.finalise_datasets` writes parsed values straight into the catalogue frame:

```python
datasets.at[idx, key] = value # mixins.py:97
```

If the column was inferred as `float64` and the file stores that attribute as a string, pandas raises rather than coercing:

```
TypeError: Invalid value '60225' for dtype 'float64'
```

## Trigger

CMIP6 data from KIOST stores `branch_time_in_child` and `branch_time_in_parent` as strings. In a DRS-ingested catalogue both columns come back `float64`, so finalising any KIOST dataset raises.

```
next institution KIOST rows 32
MISMATCH column= branch_time_in_child dtype= float64 parsed value= '60225'
MISMATCH column= branch_time_in_parent dtype= float64 parsed value= '60225'
```

## Why it matters

The exception propagates out of `DataCatalog.finalise` and kills `ref solve`. On a DRS-ingested catalogue, finalisation happens inside the first solve (see #837), so 32 files from one model abort a run that had already spent hours reading. Nothing is committed, because finalisation only writes at the end of the batch, so the work is lost too.

A single non-conforming model should not be able to end a solve over 11,000 datasets.

## Suggested fix

Coerce on assignment rather than trusting the inferred dtype. Either build the finalised frame column-wise from the parsed records and let pandas infer once, or upcast the target column to `object` when the incoming value does not fit. Failing that, catch per-dataset and mark it invalid rather than propagating, so one bad model is skipped rather than fatal.

Seen on climate-ref v0.16.2.

Contributor guide

Open the contributing guide

Research direction

Start in mixins.py:97 at FinaliseableDatasetAdapterMixin.finalise_datasets, then trace how DataCatalog.finalise is called from ref solve. Reproduce the KIOST case with the branch_time_in_child and branch_time_in_parent columns and inspect the parsed values against their inferred dtypes. Done means a non-conforming model no longer aborts the whole solve and the affected datasets are handled according to the chosen failure behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.