bcgov / bcgov/nr-rfc-processing

Integrate All Steps with Object Storage

Open
#43 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
4
Forks
2
PR merge metrics
No merged PRs in 30d

Description

When the script is run want it to be able to pickup from previous failures.

* With the download process after data is downloaded it is copied to object storage, immediately after
* When the download process starts before it attempts to retrieve data from object storage first before going to its official sources
* When processing starts, it determines what needs to be done based on what data exists in the object storage

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the download and processing flows described in the issue, including where downloaded data is copied and where processing determines its work. Done means downloads check object storage before official sources, completed downloads are stored immediately, and processing resumes based on the data present there.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.