Climate-REF / Climate-REF/climate-ref
finalise_datasets aborts the whole solve when a model stores a numeric attribute as a string
- Dominant language
- Python
- Stars
- 26
- Forks
- 14
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 36
Description
`FinaliseableDatasetAdapterMixin.finalise_datasets` writes parsed values straight into the catalogue frame:
```python
datasets.at[idx, key] = value # mixins.py:97
```
If the column was inferred as `float64` and the file stores that attribute as a string, pandas raises rather than coercing:
```
TypeError: Invalid value '60225' for dtype 'float64'
```
## Trigger
CMIP6 data from KIOST stores `branch_time_in_child` and `branch_time_in_parent` as strings. In a DRS-ingested catalogue both columns come back `float64`, so finalising any KIOST dataset raises.
```
next institution KIOST rows 32
MISMATCH column= branch_time_in_child dtype= float64 parsed value= '60225'
MISMATCH column= branch_time_in_parent dtype= float64 parsed value= '60225'
```
## Why it matters
The exception propagates out of `DataCatalog.finalise` and kills `ref solve`. On a DRS-ingested catalogue, finalisation happens inside the first solve (see #837), so 32 files from one model abort a run that had already spent hours reading. Nothing is committed, because finalisation only writes at the end of the batch, so the work is lost too.
A single non-conforming model should not be able to end a solve over 11,000 datasets.
## Suggested fix
Coerce on assignment rather than trusting the inferred dtype. Either build the finalised frame column-wise from the parsed records and let pandas infer once, or upcast the target column to `object` when the incoming value does not fit. Failing that, catch per-dataset and mark it invalid rather than propagating, so one bad model is skipped rather than fatal.
Seen on climate-ref v0.16.2.
Contributor guide
Research direction
Start in mixins.py:97 at FinaliseableDatasetAdapterMixin.finalise_datasets, then trace how DataCatalog.finalise is called from ref solve. Reproduce the KIOST case with the branch_time_in_child and branch_time_in_parent columns and inspect the parsed values against their inferred dtypes. Done means a non-conforming model no longer aborts the whole solve and the affected datasets are handled according to the chosen failure behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100