awslabs / awslabs/python-deequ

FileSystemMetricsRepository file to be a parquet (/ delta)

Open
#185 2 comments 0 reactions 0 assignees View on GitHub
enhancement feature request
Dominant language
Jupyter Notebook
Stars
826
Forks
158
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
We run thousands of anomalies validations on hundreds of data sources. We often want to explain why they fail. To do so - first step is to look at the underlying metrics - persisted in PyDeeQu managed json. Processing json in spark environment is not efficient (both programmatically and not).

**Describe the solution you'd like**
One way to provide explanations for failed anomalies is to expose json repository file underlying FileSystemMetricsRepository to analytical user. Since PyDeeQu is a PySpark framework the best and natural choice is to store repository data as .parquet/delta files instead of json. This could also cover:

- Enable anomalies explainability by enabling live metrics repository table
- Establishing data contract for repository (currently PyDeeQu json repository does not really assume a schema)

**Describe alternatives you've considered**
Consider DELTA format - probably way batter choice but requires more development to enable incremental processing and rollbacks.

**Additional context**
Would also be good to know why .json format was a chosen? And is there any way we can benefit from this choice?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.