ISISNeutronMuon / ISISNeutronMuon/analytics-data-platform

Data quality checks

Open
#361 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 7h
Merged PRs (30d)
21

Description

We are currently on the back foot if errors in the data are discovered as users spot them. We need to support more robust observability and quality checks on the pipelines. Some examples:

- ingestion errors: What went wrong with a pipeline?
- freshness: Indicators of how when data tables/models were last updated
- source errors: Has a source stopped producing data unexpectedly?
- pipeline tests (unit and data): support developers to know pipelines are correct

There is overlap with #78 but this is also a separate issue understanding the state of the data itself.

Recent examples of where this has gone wrong:

- #166, #358, #359
- #360
- #362

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the overlap with #78 and the recent failure examples in #166, #358, #359, #360, and #362. Identify which ingestion, freshness, source, and pipeline-quality signals are already supported; done requires an agreed scope and implementation plan for robust data observability and quality checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, observability-sre, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.