mapbox / mapbox/nepomuk

Dataset Validation Pipleline

Open
#120 0 comments 0 reactions 0 assignees View on GitHub
Backend Discussion Feature Request Starter Task
Dominant language
C++
Stars
24
Forks
7
PR merge metrics
No merged PRs in 30d

Description

GTFS datasets are a tool that allows very diverse specifications. When importing new datasets, we should ensure that these datasets hold up to our expectations.
Some of the tests we should specify are simply relying on using available tools, others are about data analysis.

## Valid GTFS

There is a wide range of tools available. We should run at least one of these tools to ensure that every dataset we import is valid GTFS and does not include strange contraptions that could implement a security risk. This part can be considered a very simple starter task.

## GTFS Sanity

We should implement checkers that allow finding issues in GTFS feeds (e.g. unspecified IDs, broken geometry...)

## Data normalisation

We should build a set of detectors that gives a heads up if certain strings do not follow expected formats. (see #52). In case of these triggers, we can start up a pre-processing to normalise strings. Other normalisation might be required as well (consecutive IDs: already done). Normalisation can be detected/done in any language as a separate tool.

## Connectivity

Some GTFS feeds do not provide the connectivity we would expect. We already offer strongly connected components. In this feature we should detect how often close stations do not allow transfers and augment the feed accordingly.

----
This list is not complete and will be augmented accordingly.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or concrete entry points are named. Begin by locating the existing GTFS import and strongly connected components functionality, then break the proposal into separately testable validation, sanity, normalization, and connectivity tasks with explicit completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.