MetOffice / MetOffice/CDDS

Extract: Skip streams if all files are present.

Open
#543 0 comments 0 reactions 0 assignees View on GitHub
migrated
Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 15h
Merged PRs (30d)
23

Description

Migrated Jira Issue CDDSO-89
Jira creation date 22/04/2022 13:47
-------
When re-running extract, following failure, if the same set of streams are used it is possible for extract to spend a long time (several hours in the case of the piControl) attempting to "fill in gaps". This is inefficient in both our time and load on MASS resources.

I propose the following solution; At the start of processing a stream/substream, extract checks for existing files for that stream/substream and if all expected files are present skips on to the next.
-------

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the extract entry point that processes streams and substreams, then trace how expected output files are identified after a failed run. Confirm how the current gap-filling work is triggered and identify the relevant tests, if any. Done means a rerun skips a stream or substream only when all expected files are already present.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.