Efficient Way to Pre-Download ERA5 inputs and Load them for Fine-Tuning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 174
- PR merge metrics
- No merged PRs in 30d
Description
I’m building on top of Aurora and adding custom decoders to predict additional variables. To do this, I still need to feed ERA5 data into Aurora to generate the latent representation before passing it to the custom decoders.
The documentation shows how to download ERA5 data for a specific day. However, what’s the best practice for pre-downloading all required ERA5 data so it can be efficiently loaded during model training, without becoming a bottleneck?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked example_era5 documentation and trace the ERA5 loading path used to feed Aurora before custom decoders. Done means documenting a reproducible way to pre-download and load the required inputs efficiently during fine-tuning, including how to avoid a training bottleneck.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100