google-research / google-research/flood-forecasting

Design target-free inference without observation files

Open
#267 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
343
Forks
87
Avg merge
14d 23m
Merged PRs (30d)
1

Description

## Design question

Should `run infer` support predictor-complete basins that have no target/observation file at all?

The current MultiMet loading path retains configured `target_variables` in inference mode and `_load_data()` unconditionally calls `_load_target_features()`. A predictor-complete synthetic inference fixture therefore fails with `FileNotFoundError` before prediction when the target tree is absent.

I am treating this as a design gap/enhancement rather than a confirmed loader regression: the current documentation does not clearly promise target-file-free inference, and supporting it affects more than dataset loading.

## Proposed behavior for discussion

1. Evaluation and fine-tuning continue to require targets.
2. Inference can omit target loading when observation files are unavailable.
3. Inference with supplied targets retains observations and current diagnostic outputs.
4. Sample validation does not require target values in target-free mode.
5. Losses and observation-dependent metrics are skipped when targets are absent.
6. Prediction coordinates and serialization remain identical in both modes.

## Questions

- Should this be automatic when target files are missing, or controlled by an explicit configuration flag?
- Should `target_variables` remain configured to preserve checkpoint/model output semantics even when no target data is loaded?
- What tester/result schema should represent unavailable observations, losses, and metrics?
- Are all-missing placeholder target files the intended current workflow instead?

If the intended behavior is confirmed, I can prepare a separate integration-tested PR covering the dataset, sample validation, tester outputs, and metrics. I would keep it independent from the CSV schema fixes in #265 and #266.

Contributor guide

Open the contributing guide

Research direction

Start with the MultiMet loading path, especially _load_data() and _load_target_features(), then trace dataset loading, sample validation, tester outputs, and metrics used by run infer. Resolve whether target-free inference is supported and how missing observations, losses, and metrics should be represented. Done requires an agreed design and integration tests covering inference with and without targets while preserving prediction coordinates and serialization.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.