NaN values in WRF pressure levels cause invalid outputs in Aurora
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 174
- PR merge metrics
- No merged PRs in 30d
Description
I am using WRF model data as input for Aurora (AI-based weather forecasting).
WRF provides all the required variables in .nc format (q.nc, t.nc, 10u.nc, 10v.nc, slt.nc, msl.nc, z_statica.nc, etc.).
Preprocessing steps performed:
- Cleaned the datasets
- Renamed variables to match Aurora naming conventions
- Converted units to match Aurora’s requirements
- Removed unnecessary dimensions
- Followed all conventions from Aurora’s online documentation
I performed regridding from the WRF grid to Aurora’s grid using:
cdo remapnn,gridspec.txt clean_data/z_static.nc interpolated_data/z_static.nc
gridspec.txt:
gridtype=lonlat
xfirst=10.23557
xinc=0.1
xsize=56
yfirst=44.77918
yinc=0.1
ysize=33
PROBLEM:
When running Aurora to predict hour 18 using hours 6 and 12 as input, all output variables are set to 9e+36.
Suspected cause:
Many variables are on a pressure (hPa) vertical scale with 7 levels: 1000, 925, 850, 700, 500, 300, 100.
For higher pressure levels (1000, 925), data is missing for a large portion of the grid because those pressure levels physically do not exist in certain cells (e.g., mountainous terrain).
In those cases, the lowest available level can be 700 hPa or above.
In the NetCDF files, these appear as NaN values (displayed as 9e+36 in ncview).
These NaNs seem to propagate through Aurora’s internal calculations, producing invalid outputs without explicit errors.
Missing data statistics (example from q.nc):
Level 1000.0 hPa: 76.95% missing values
Level 925.0 hPa: 46.40% missing values
Level 850.0 hPa: 19.47% missing values
Level 700.0 hPa: 0.21% missing values
Level 500.0 hPa: 0.00% missing values
Level 300.0 hPa: 0.00% missing values
Level 100.0 hPa: 0.00% missing values
Question:
What is the recommended way to handle these physically unavoidable NaN values in Aurora input data?
I would like to avoid removing the pressure levels with high NaN percentages, since they are close to the ground and critical for forecasting severe weather events (e.g., thunderstorms).
Can you direct me to parts of the documentation that discuss this, if they exist? I haven't found them.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Aurora’s online documentation and the WRF NetCDF inputs, especially q.nc and the pressure-level files, then review the reported CDO remapnn command and gridspec.txt. Determine whether the documentation specifies handling for physically missing pressure-level values and whether Aurora reports or validates them. Done means a documented, supported recommendation for these NaNs or a clearly identified documentation gap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100