google-deepmind / google-deepmind/weathernext
Clarification on Global Mean RMSE Oscillation for Near-Surface Temperatures
- Dominant language
- Python
- Stars
- 7.7k
- Forks
- 986
- PR merge metrics
- No merged PRs in 30d
Description
Hi
I’m observing an interesting behavior while evaluating GraphCast forecasts against ERA5 data.
When plotting the global mean of near-surface temperature variables, specifically t1000 and t2m, the forecast mean oscillates in sync with ERA5, which looks physically correct and expected.
However, the global mean RMSE computed for a single 40-step (6-hour interval) run also shows a noticeable oscillatory pattern instead of a smooth or gradually increasing trend.
Is this RMSE oscillation expected, or does it indicate an underlying issue with how RMSE is being computed or with the model’s short-term stability?
Details:
Model: GraphCast-Small (1.0° / 13 levels)
Forecast length: 40 steps (6-hour intervals)
RMSE computed as global cosine-latitude-weighted mean over all grid points and times
Both forecast and ERA5 data are aligned on valid time coordinates before evaluation
RMSE typically increases smoothly with lead time, so I wanted to confirm whether oscillations in RMSE are normal behavior for GraphCast for near-surface temperature variables.
Contributor guide
Research direction
No source files or tests are named. Start by locating the evaluation entry point for GraphCast-Small forecasts and reproduce the 40-step t1000 and t2m comparison using the stated valid-time alignment and cosine-latitude weighting. Done means determining whether the oscillation comes from the RMSE calculation or documented model behavior and recording the evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100