microsoft / microsoft/aurora

Aurora air pollution model predicts NaNs / exhausts after around 13 days of autoregressive rollout

Open
#183 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
174
PR merge metrics
No merged PRs in 30d

Description

Hi,

I am running the Aurora air pollution model autoregressively and I am seeing that the model becomes unstable after about 13 days of forecasting. After that point, the predictions start to degrade badly and eventually produce NaN values.

What I observed
The rollout is stable initially.
Around day 13, the forecast starts to drift strongly.
After that, the model appears to exhaust / blow up and begins predicting NaNs.
In my tests, the surface variables degrade much faster, while some pressure-level variables remain relatively reasonable for longer.
My question

Is this expected behavior for the Aurora air pollution model during long autoregressive rollouts?

Since the model is mainly designed for shorter forecasts, I would like to understand:
Is there a known reason the model exhausts during longer rollouts?
What are the recommended ways to extend the forecast horizon more safely?

Specifically, I would appreciate guidance on:
Whether this is a limitation of the released checkpoint
Whether any normalization or preprocessing issues could cause this
Whether clamping is recommended
Best practices for extending inference beyond the standard forecast range

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by reproducing the reported autoregressive Aurora air-pollution rollout through the point where NaNs appear, then inspect the inference and normalization paths involved. Done means identifying whether the behavior is expected or a defect and documenting supported practices for longer forecasts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.