Switch to using queuing system for dd.weather.gc.ca
- Dominant language
- Python
- Stars
- 65
- Forks
- 11
- Avg merge
- 21h 25m
- Merged PRs (30d)
- 70
Description
**Describe the task**
Switch to using AMQP for environment Canada data.
**Acceptance Criteria**
- [ ] Using AMQP instead of polling.
- [ ] Long running pod that processes data as and when it becomes available.
**Additional context**
- The recommended approach for getting grib files is using the AMQP queue. We were blocked from using it when we originally developer our app, because that port was blocked.
- We're currently polling the environment canada site using cronjobs and then downloading new information when found. This introduces a lag between when the data is available, and when we process it.
- https://eccc-msc.github.io/open-data/msc-datamart/readme_en/
Contributor guide
Research direction
Start by tracing the current cronjob polling and Environment Canada download flow, then read the linked ECCC AMQP documentation. Replace the polling path with an AMQP consumer that processes GRIB files as they become available and runs in a long-lived pod; done means polling is no longer required and the pod continuously processes new data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100