MetOffice / MetOffice/CDDS

Allow extract validate to check a subset of data files

Open
#1,033 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 15h
Merged PRs (30d)
23

Description

It may be valuable to allow extract validate to check a subset of input data files (e.g. the first 10 years) if the amount of data files is otherwise too large (e.g. files spanning longer than 50 years) . This would improve efficiency and allow for some level of validation even for the largest of data sets.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the `extract validate` command entry point and reading how it currently selects and checks input data files. Clarify the intended subset selector, such as a year range or file count, and define done as validating only that selected subset while preserving current full-dataset validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.