Extract: Skip streams if all files are present.
- Dominant language
- Python
- Stars
- 8
- Forks
- 7
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 23
Description
Migrated Jira Issue CDDSO-89
Jira creation date 22/04/2022 13:47
-------
When re-running extract, following failure, if the same set of streams are used it is possible for extract to spend a long time (several hours in the case of the piControl) attempting to "fill in gaps". This is inefficient in both our time and load on MASS resources.
I propose the following solution; At the start of processing a stream/substream, extract checks for existing files for that stream/substream and if all expected files are present skips on to the next.
-------
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the extract entry point that processes streams and substreams, then trace how expected output files are identified after a failed run. Confirm how the current gap-filling work is triggered and identify the relevant tests, if any. Done means a rerun skips a stream or substream only when all expected files are already present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100