openclimatefix / openclimatefix/nwp-consumer
Make consumer retry earlier init_times if the current one i now available
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17
- Forks
- 10
- Avg merge
- 1h 46m
- Merged PRs (30d)
- 1
Description
When UKV service is severely delayed, sometimes it has already started returning earlier init-times, but we still wouldn't download them because the consumer is looking for the latest one, which is still delayed. For example:
- The consumer has downloaded the 6 am init-time
- UKV started having issues after that, and 9 am init-time came in at 5 pm. Later init-time started coming in after that, but the service is still delayed several hours for "current" init-time
- Consumer continuously fails, as by now it's trying to download the 21:00 run which is delayed, and is not looking for the 9:00, 12:00, 15:00 etc that are now available
- Day Ahead model starts running out of UKV data at the tail end and can't rerun, intraday forecast is running on data 12h+ old
(This is what happened yesterday)
One thing I thought could solve this, is if after not being able to find the current init-time the consumer would look for the one before, and the one before, and so on, until it gets to the last one it has downloaded
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named. Locate the consumer logic that selects the current init-time, then trace how unavailable runs are handled; done means it checks earlier init-times down to the last downloaded one and adds coverage for the delayed-service scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100