microsoft / microsoft/aurora

NaN values in WRF pressure levels cause invalid outputs in Aurora

Open
#137 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
174
PR merge metrics
No merged PRs in 30d

Description

I am using WRF model data as input for Aurora (AI-based weather forecasting).
WRF provides all the required variables in .nc format (q.nc, t.nc, 10u.nc, 10v.nc, slt.nc, msl.nc, z_statica.nc, etc.).
Preprocessing steps performed:

  • Cleaned the datasets
  • Renamed variables to match Aurora naming conventions
  • Converted units to match Aurora’s requirements
  • Removed unnecessary dimensions
  • Followed all conventions from Aurora’s online documentation
    I performed regridding from the WRF grid to Aurora’s grid using:
    cdo remapnn,gridspec.txt clean_data/z_static.nc interpolated_data/z_static.nc
    gridspec.txt:
    gridtype=lonlat
    xfirst=10.23557
    xinc=0.1
    xsize=56
    yfirst=44.77918
    yinc=0.1
    ysize=33

PROBLEM:
When running Aurora to predict hour 18 using hours 6 and 12 as input, all output variables are set to 9e+36.

Suspected cause:
Many variables are on a pressure (hPa) vertical scale with 7 levels: 1000, 925, 850, 700, 500, 300, 100.
For higher pressure levels (1000, 925), data is missing for a large portion of the grid because those pressure levels physically do not exist in certain cells (e.g., mountainous terrain).
In those cases, the lowest available level can be 700 hPa or above.
In the NetCDF files, these appear as NaN values (displayed as 9e+36 in ncview).

These NaNs seem to propagate through Aurora’s internal calculations, producing invalid outputs without explicit errors.

Missing data statistics (example from q.nc):
Level 1000.0 hPa: 76.95% missing values
Level 925.0 hPa: 46.40% missing values
Level 850.0 hPa: 19.47% missing values
Level 700.0 hPa: 0.21% missing values
Level 500.0 hPa: 0.00% missing values
Level 300.0 hPa: 0.00% missing values
Level 100.0 hPa: 0.00% missing values

Question:
What is the recommended way to handle these physically unavoidable NaN values in Aurora input data?
I would like to avoid removing the pressure levels with high NaN percentages, since they are close to the ground and critical for forecasting severe weather events (e.g., thunderstorms).
Can you direct me to parts of the documentation that discuss this, if they exist? I haven't found them.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Aurora’s online documentation and the WRF NetCDF inputs, especially q.nc and the pressure-level files, then review the reported CDO remapnn command and gridspec.txt. Determine whether the documentation specifies handling for physically missing pressure-level values and whether Aurora reports or validates them. Done means a documented, supported recommendation for these NaNs or a clearly identified documentation gap.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.