awslabs / awslabs/python-deequ
Enable Custom Analyzer in python by implementing pydeequ.analyzers._AnalyzerObject
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Enabling and showing the example of how to extend pydeequ.analyzers._AnalyzerObject to define custom Analyzer in python.
**Describe the solution you'd like**
Be able to implement:
```
class MyCustomAnalyzer(_AnalyzerObject):
"""Get the maximum of a numeric column."""
def __init__(self, column, my_property: str = None):
"""
:param str column: column to find the maximum.
:param str my_property: custom property
"""
self.column = column
self.my_property = my_property
@property
def _analyzer_jvm(self, foo: AnalyzerInput) -> AnalyzerOutput:
# my custom transformation that transforms well defined AnalyzerInput into AnalyzerOutput
bar: AnalyzerOutput = ...
return bar
```
and then run it in VerificationSuite, ex:
```
results = (VerificationSuite(spark)
.onData(df)
.useRepository(repository)
.saveOrAppendResult(ResultKey(spark, ResultKey.current_milli_time(), {'tag': 'my-tag'}))
.addAnomalyCheck(OnlineNormalStrategy(
lowerDeviationFactor=0.01,
upperDeviationFactor=0.01,
ignoreStartPercentage=0.1,
ignoreAnomalies=False,
), MyCustomAnalyzer("column_name", my_property="yeey!"))
.run())
```
**Describe alternatives you've considered**
When calculating Anomalies - every time I have a custom metrics (to focus attention - lets say `Sum() / CountDistinct()`) I build temporary table that has one row, ex:
```
| value_unique_name |
-----------------------------------
|
Contributor guide
Assessment
This issue has not been assessed yet.