ISISNeutronMuon / ISISNeutronMuon/analytics-data-platform

Create a structure for running automated tests on ELT pipelines

Open
#443 2 comments 0 reactions 1 assignee Claimed by @WHTaylor View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
2d 7h
Merged PRs (30d)
21

Description

When either a new ELT pipeline is written or changes are made to an existing pipeline, the onus is on the reviewer to check the changes execute successfully. This is not a sustainable approach and certainly not up to modern best practice where we would expect CI testing to happen as standard.

The aim of this issue is to define a structure within `elt-pipelines` for automated testing of the pipelines. As discussed, we'll start by using the real sources and have them write to a local, SQLite-based, catalog and have some mechanism for asserting the state of the loaded tables. Part of the work here is to think how that should be implemented. We considered mocking the sources but this could get very involved for complex sources and would need to evolve inline with the data sources themselves for the tests not to hide actual errors when the pipelines run in production.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.