awslabs / awslabs/python-deequ

Enable Custom Analyzer in python by implementing pydeequ.analyzers._AnalyzerObject

Open
#186 1 comment 0 reactions 0 assignees View on GitHub
enhancement feature request
Dominant language
Jupyter Notebook
Stars
826
Forks
158
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Enabling and showing the example of how to extend pydeequ.analyzers._AnalyzerObject to define custom Analyzer in python.

**Describe the solution you'd like**
Be able to implement:

```
class MyCustomAnalyzer(_AnalyzerObject):
"""Get the maximum of a numeric column."""

def __init__(self, column, my_property: str = None):
"""
:param str column: column to find the maximum.
:param str my_property: custom property
"""
self.column = column
self.my_property = my_property

@property
def _analyzer_jvm(self, foo: AnalyzerInput) -> AnalyzerOutput:
# my custom transformation that transforms well defined AnalyzerInput into AnalyzerOutput
bar: AnalyzerOutput = ...
return bar

```
and then run it in VerificationSuite, ex:
```
results = (VerificationSuite(spark)
.onData(df)
.useRepository(repository)
.saveOrAppendResult(ResultKey(spark, ResultKey.current_milli_time(), {'tag': 'my-tag'}))
.addAnomalyCheck(OnlineNormalStrategy(
lowerDeviationFactor=0.01,
upperDeviationFactor=0.01,
ignoreStartPercentage=0.1,
ignoreAnomalies=False,
), MyCustomAnalyzer("column_name", my_property="yeey!"))
.run())
```

**Describe alternatives you've considered**
When calculating Anomalies - every time I have a custom metrics (to focus attention - lets say `Sum() / CountDistinct()`) I build temporary table that has one row, ex:

```
| value_unique_name |
-----------------------------------
|

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.