Dataset Validation Pipleline
- Dominant language
- C++
- Stars
- 24
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
GTFS datasets are a tool that allows very diverse specifications. When importing new datasets, we should ensure that these datasets hold up to our expectations.
Some of the tests we should specify are simply relying on using available tools, others are about data analysis.
## Valid GTFS
There is a wide range of tools available. We should run at least one of these tools to ensure that every dataset we import is valid GTFS and does not include strange contraptions that could implement a security risk. This part can be considered a very simple starter task.
## GTFS Sanity
We should implement checkers that allow finding issues in GTFS feeds (e.g. unspecified IDs, broken geometry...)
## Data normalisation
We should build a set of detectors that gives a heads up if certain strings do not follow expected formats. (see #52). In case of these triggers, we can start up a pre-processing to normalise strings. Other normalisation might be required as well (consecutive IDs: already done). Normalisation can be detected/done in any language as a separate tool.
## Connectivity
Some GTFS feeds do not provide the connectivity we would expect. We already offer strongly connected components. In this feature we should detect how often close stations do not allow transfers and augment the feed accordingly.
----
This list is not complete and will be augmented accordingly.
Contributor guide
Research direction
No files, tests, or concrete entry points are named. Begin by locating the existing GTFS import and strongly connected components functionality, then break the proposal into separately testable validation, sanity, normalization, and connectivity tasks with explicit completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100