bcgov / bcgov/nr-rfc-processing

Increase Script / Data reliability

Open
#62 0 comments 0 reactions 0 assignees View on GitHub
Epic
Dominant language
Python
Stars
4
Forks
2
PR merge metrics
No merged PRs in 30d

Description

At the moment the script is setup to run a single date. Sometimes this doesn't work as the modis data or the viirs data is not available when the script runs.

This epic will implement the following:
* when the pipeline runs it will evaluate what data we already have and the data that is available and process based on that configuration
* Separate the steps taken in the dailypipeline so that if one step fails it doesn't impact other unrelated scripts.
* Reconfigure scripts so that they can automatically pickup where they last got to.
* when data is downloaded immediately push it back to object storage.
* before downloading data check to see if the data exists in ostore first
* general improvements to try to make the code more readable. (use utility methods with descriptive names vs file name / date / data hacking using string manipulation regex's etc.) Methods should also provide clear description of what they do with examples

# Dependencies

## Epic

- [ ] bcgov/nr-rfc-processing/issues/44
- [ ] bcgov/nr-rfc-processing/issues/60
- [ ] bcgov/nr-rfc-processing/issues/63
- [ ] bcgov/nr-rfc-processing/issues/67

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the dailypipeline entry point and review the dependency issues 44, 60, 63, and 67. Trace the existing data download and processing steps, then define completion around availability-aware processing, isolated failures, resumable progress, and object-storage checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.