bcgov / bcgov/wps

Cloud API for Weather Data Sharing & Integrations

Open
#3,760 3 comments 0 reactions 0 assignees View on GitHub
4Refinement Epic
Dominant language
Python
Stars
65
Forks
11
Avg merge
21h 25m
Merged PRs (30d)
70

Description

**Target Group:**
Industry Researchers, Partners, and Interested Parties who want to consume BCWS Weather Data

**Focus:**
Our focus is making BCWS Weather Data publicly available through a public API that is accessible, interactive, and offers a programmable method for querying and researching data;
We believe this will deliver an improved and modern method for consuming and monitoring this data source;
This will be confirmed when:
a) We can shut down CSV files on the datamart ftp site
b) We receive positive feedback from our end users
c) The BCWS Predictive Services inbox no longer receives technical support questions from the public and stakeholders regarding how to access weather data.

**Proposed Solution**
https://developer.gov.bc.ca/docs/default/component/aps-infra-platform-docs/

https://developer.gov.bc.ca/docs/default/component/aps-infra-platform-docs/tutorials/quick-start/

https://chat.developer.gov.bc.ca/channel/aps-ops

**Product Goals in order of Priority:**
1. No performance issues w/ operational database when users are interacting with API. Some users may be accessing hourly weather data for a subset of stations every hour. Other users may wish to access the entire historical database for all stations in the province.
2. System notifies file in ftp or similar when huge data packages are ready after a complicated/huge query.
3. User documentation is available.
4. Advanced search/query parameters: Search by station id or station name; by date range; within a defined geographic extent; for values above OR below a weather/FWI threshold.
5. Query station data from an interactive map.

**Key Performance Indicators**
KPI1 = Eliminate Throttling (current limitation is 365 records at any time otherwise the system will shut down access to ALL users)
KPI2 = Strong UX & Documentation to eliminate support calls/emails
KPI3 = PO to provide current cost/effort via Vendor: cost benefits guidance on current solution development + maintenance cost, cost to make updates, troubleshooting costs

**Cadillac Version (out of scope)**
Landing page for subscription updates/change notifications

**Additional Context**
- The data that need to be exposed are stored in a Oracle Database on an NRIDS server and managed by Vivid. The main Oracle server alias is db-wfoprd1.nrs.bcgov. The main server is mirrored to a secondary alias (db-wfoprd2.nrs.bcgov) when higher load is placed on the main server. Read-only traffic should be using the secondary alias.
- A copy of these data are also stored on a public FTP site as CSV files here: https://www.for.gov.bc.ca/ftp/HPR/external/!publish/BCWS_DATA_MART/
- In the future, a cloud database will need to automatically mirror the operational Oracle Database.
- This proof of concept could use CSV files from the public FTP site to create a static cloud database and an associated API. Mirroring the operational database could happen at a later time.

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or entry points are named. Start by reviewing the proposed API-platform documentation and the existing BCWS CSV data on the public FTP site, then determine the scope for exposing Oracle weather data or a static cloud database. Done would require a defined API with safe query performance, large-query handling, user documentation, and the stated support and CSV-retirement outcomes.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, postgresql, python
Domain
api, backend, cloud, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.