bcgov / bcgov/wps

ASA : Data Model Architecture

Open
#2,251 1 comment 0 reactions 0 assignees View on GitHub
Task
Dominant language
Python
Stars
65
Forks
11
Avg merge
21h 25m
Merged PRs (30d)
70

Description

**Describe the task**
Note: Deferring for now... going to take prototype a bit further.

As we've prototyped solutions for different features, we've learned a lot about what's possible with tools in the GIS domain such as GDAL and postgis. It's also led to some disparate data sources that are unclear on their boundaries and purpose explicitly.

For different use-cases we've stored data in different places and in different ways, currently:
- S3 hosts the generated SFMS raster data that we read in on the fly for displaying raw HFI, slope, aspect, elevation from a raster server
- Our `tileserver` database/API has tables for fire zone and fire centre polygons that we serve up for displaying / analyzing fire zones/centres across the province, as well as a HFI table that stores calculated HFI areas
- Our `wps` API database stores advisory metadata based on offline advisory calculations depending on lookup intersections on the HFI table in the `tileserver`

**Acceptance Criteria**
- [ ] Schedule Architectural discussion
- [ ] Determine data source boundaries for types of workloads features (analysis database, raster database, S3 object store) and document it in our data architecture diagram and design doc
- [ ] Agree if we need more explicit naming in our `wps` API database, and if we should pull the desired analysis tables into it's own database
- [ ] Address any unclear conceptual understandings from developers so that we get ahead of mistakes that could affect velocity/understanding in moving advisory work forward

- [ ] discuss idea of creating separate DB for ASA to increase resiliency
- [ ] rethink how/where data for ASA gets stored in our regular (non-tileserv) DB
- [ ] is there anything else we could/should have done to prevent this failure from happening?

**Additional context**
- Current architecture diagram: https://github.com/bcgov/wps/blob/main/docs/ARCH.md
- May not be able to consider this without also re-evaluating larger architecture and project structure.
- Implementing changes in Architecture in separate ticket
- Diagram in a separate ticket

Contributor guide

Open the contributing guide

Research direction

Start by reading docs/ARCH.md and reviewing the current S3, tileserver, and wps data boundaries described in the issue. The work is deferred and depends on an architectural discussion and separate implementation and diagram tickets. Done means agreed workload boundaries, database naming and resilience decisions, and updates to the architecture documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
data, databases, documentation
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.