ISISNeutronMuon / ISISNeutronMuon/analytics-data-platform
Data quality checks
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 21
Description
We are currently on the back foot if errors in the data are discovered as users spot them. We need to support more robust observability and quality checks on the pipelines. Some examples:
- ingestion errors: What went wrong with a pipeline?
- freshness: Indicators of how when data tables/models were last updated
- source errors: Has a source stopped producing data unexpectedly?
- pipeline tests (unit and data): support developers to know pipelines are correct
There is overlap with #78 but this is also a separate issue understanding the state of the data itself.
Recent examples of where this has gone wrong:
- #166, #358, #359
- #360
- #362
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the overlap with #78 and the recent failure examples in #166, #358, #359, #360, and #362. Identify which ingestion, freshness, source, and pipeline-quality signals are already supported; done requires an agreed scope and implementation plan for robust data observability and quality checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, observability-sre, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100