awslabs / awslabs/python-deequ
Retrieving indication of failed metrics
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- Avg merge
- 9d 22h
- Merged PRs (30d)
- 3
Description
**Is your feature request related to a problem? Please describe.**
I have a process that computes data metrics using deequ. I noticed that when a spark job computing these metrics fails, they are stored as failed metrics in the AnalyzerContext Scala object. However, the only API I seem to find for retrieving the result metrics is `successMetricsAsDataFrame `or `successMetricsAsJson` which both call the Scala APIs that filter out the failed metrics. This means that I have no simple way of finding out if metrics failed due to Spark job failure in order to rerun it or investigate the reasons for the failure.
**Describe the solution you'd like**
Ideally a translation of the metric map in AnalyzerContext into a python object. This would also solve the failed metric problem and also I won't have to manually parse the results from the dataframe in order to save them in the format I would like to. Another possibility would be to add a `allMetrics` getter to the API.
**Additional context**
[The python AnalyzerContext object](https://github.com/awslabs/python-deequ/blob/7ec9f6f72839779f370a4753c0db26d6cf052203/pydeequ/analyzers.py#L55)
[The scala AnalyzerContext object](https://github.com/awslabs/deequ/blob/933417676189bc7833166f976fd024a4b2177292/src/main/scala/com/amazon/deequ/analyzers/runners/AnalyzerContext.scala#L48)
Contributor guide
Assessment
This issue has not been assessed yet.