bcgov / bcgov/wps

Switch to using queuing system for dd.weather.gc.ca

Open
#1,428 2 comments 0 reactions 0 assignees View on GitHub
4Refinement Task
Dominant language
Python
Stars
65
Forks
11
Avg merge
21h 25m
Merged PRs (30d)
70

Description

**Describe the task**
Switch to using AMQP for environment Canada data.

**Acceptance Criteria**
- [ ] Using AMQP instead of polling.
- [ ] Long running pod that processes data as and when it becomes available.

**Additional context**
- The recommended approach for getting grib files is using the AMQP queue. We were blocked from using it when we originally developer our app, because that port was blocked.
- We're currently polling the environment canada site using cronjobs and then downloading new information when found. This introduces a lag between when the data is available, and when we process it.
- https://eccc-msc.github.io/open-data/msc-datamart/readme_en/

Contributor guide

Open the contributing guide

Research direction

Start by tracing the current cronjob polling and Environment Canada download flow, then read the linked ECCC AMQP documentation. Replace the polling path with an AMQP consumer that processes GRIB files as they become available and runs in a long-lived pod; done means polling is no longer required and the pod continuously processes new data.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
backend, data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.