google-deepmind / google-deepmind/weathernext

Clarification on Global Mean RMSE Oscillation for Near-Surface Temperatures

Open
#188 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.7k
Forks
986
PR merge metrics
No merged PRs in 30d

Description

Hi
I’m observing an interesting behavior while evaluating GraphCast forecasts against ERA5 data.

When plotting the global mean of near-surface temperature variables, specifically t1000 and t2m, the forecast mean oscillates in sync with ERA5, which looks physically correct and expected.

However, the global mean RMSE computed for a single 40-step (6-hour interval) run also shows a noticeable oscillatory pattern instead of a smooth or gradually increasing trend.

Is this RMSE oscillation expected, or does it indicate an underlying issue with how RMSE is being computed or with the model’s short-term stability?

Details:
Model: GraphCast-Small (1.0° / 13 levels)
Forecast length: 40 steps (6-hour intervals)
RMSE computed as global cosine-latitude-weighted mean over all grid points and times
Both forecast and ERA5 data are aligned on valid time coordinates before evaluation

RMSE typically increases smoothly with lead time, so I wanted to confirm whether oscillations in RMSE are normal behavior for GraphCast for near-surface temperature variables.

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by locating the evaluation entry point for GraphCast-Small forecasts and reproduce the 40-step t1000 and t2m comparison using the stated valid-time alignment and cosine-latitude weighting. Done means determining whether the oscillation comes from the RMSE calculation or documented model behavior and recording the evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.