awslabs / awslabs/python-deequ

Retrieving indication of failed metrics

Open
#109 1 comment 0 reactions 0 assignees View on GitHub
enhancement feature request
Dominant language
Jupyter Notebook
Stars
826
Forks
158
Avg merge
9d 22h
Merged PRs (30d)
3

Description

**Is your feature request related to a problem? Please describe.**
I have a process that computes data metrics using deequ. I noticed that when a spark job computing these metrics fails, they are stored as failed metrics in the AnalyzerContext Scala object. However, the only API I seem to find for retrieving the result metrics is `successMetricsAsDataFrame `or `successMetricsAsJson` which both call the Scala APIs that filter out the failed metrics. This means that I have no simple way of finding out if metrics failed due to Spark job failure in order to rerun it or investigate the reasons for the failure.

**Describe the solution you'd like**
Ideally a translation of the metric map in AnalyzerContext into a python object. This would also solve the failed metric problem and also I won't have to manually parse the results from the dataframe in order to save them in the format I would like to. Another possibility would be to add a `allMetrics` getter to the API.

**Additional context**
[The python AnalyzerContext object](https://github.com/awslabs/python-deequ/blob/7ec9f6f72839779f370a4753c0db26d6cf052203/pydeequ/analyzers.py#L55)

[The scala AnalyzerContext object](https://github.com/awslabs/deequ/blob/933417676189bc7833166f976fd024a4b2177292/src/main/scala/com/amazon/deequ/analyzers/runners/AnalyzerContext.scala#L48)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.