Allow extract validate to check a subset of data files
- Dominant language
- Python
- Stars
- 8
- Forks
- 7
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 23
Description
It may be valuable to allow extract validate to check a subset of input data files (e.g. the first 10 years) if the amount of data files is otherwise too large (e.g. files spanning longer than 50 years) . This would improve efficiency and allow for some level of validation even for the largest of data sets.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the `extract validate` command entry point and reading how it currently selects and checks input data files. Clarify the intended subset selector, such as a year range or file count, and define done as validating only that selected subset while preserving current full-dataset validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100