bcgov / bcgov/nr-rfc-processing
Increase Script / Data reliability
- Dominant language
- Python
- Stars
- 4
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
At the moment the script is setup to run a single date. Sometimes this doesn't work as the modis data or the viirs data is not available when the script runs.
This epic will implement the following:
* when the pipeline runs it will evaluate what data we already have and the data that is available and process based on that configuration
* Separate the steps taken in the dailypipeline so that if one step fails it doesn't impact other unrelated scripts.
* Reconfigure scripts so that they can automatically pickup where they last got to.
* when data is downloaded immediately push it back to object storage.
* before downloading data check to see if the data exists in ostore first
* general improvements to try to make the code more readable. (use utility methods with descriptive names vs file name / date / data hacking using string manipulation regex's etc.) Methods should also provide clear description of what they do with examples
# Dependencies
## Epic
- [ ] bcgov/nr-rfc-processing/issues/44
- [ ] bcgov/nr-rfc-processing/issues/60
- [ ] bcgov/nr-rfc-processing/issues/63
- [ ] bcgov/nr-rfc-processing/issues/67
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the dailypipeline entry point and review the dependency issues 44, 60, 63, and 67. Trace the existing data download and processing steps, then define completion around availability-aware processing, isolated failures, resumable progress, and object-storage checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100