microsoft / microsoft/aurora

Issue on checkpoints

Open
#53 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
174
PR merge metrics
No merged PRs in 30d

Description

Thanks for sharing this project. It is great to see such a powerful foundation model!

I have been testing the open-source model and the corresponding checkpoints provided by your team in ERA5. We used the ERA5 data of May 2024 as input and followed the steps in https://microsoft.github.io/aurora/example_era5.html. We found that RMSE was similar to IFS, but there was a gap between GraphCast. The results from our experiments have not been as promising, and we are unsure if this might be due to the checkpoint we are using or if there are specific aspects of the model or experimental setup that we may have overlooked.

Could you kindly confirm whether the checkpoint we are using corresponds to the model that delivered the best results in your published work? If not, are there other checkpoints or configurations that we should consider to replicate the performance presented in the paper? Additionally, any insights on potential pitfalls or nuances in the implementation would be greatly appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ERA5 example at https://microsoft.github.io/aurora/example_era5.html and identify which checkpoint and configuration were used. Compare those details with the checkpoint associated with the published results, then document whether the setup can reproduce the reported GraphCast comparison or whether another checkpoint is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.