bcgov / bcgov/nr-rfc-processing

Modify Daily pipeline so fills in missing data for a given time window

Open
#44 1 comment 0 reactions 1 assignee Claimed by @franTarkenton View on GitHub
bug enhancement
Dominant language
Python
Stars
4
Forks
2
PR merge metrics
No merged PRs in 30d

Description

currently the script is setup to run on a daily basis where it runs using the current date.

Issues with this approach is the data is not always available for the current date. Sometimes there can be a 3-4 day lag before data becomes available.

Changes:
* instead of just running a specific date, the script will check what data exists in the object store bucket vs the data that is currently available. It will then run the pipeline for consecutive days, untill al the data that is currently available has been processed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.