awslabs / awslabs/python-deequ
How do I use the binningUdf parameter
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- Avg merge
- 9d 22h
- Merged PRs (30d)
- 3
Description
I am not able to use the [`binningUdf` parameter of the Histogram analyzer](https://github.com/awslabs/python-deequ/blob/master/pydeequ/analyzers.py#L435), it errors when performing this [line](https://github.com/awslabs/python-deequ/blob/master/pydeequ/analyzers.py#L462), getting error message:
```
AttributeError Traceback (most recent call last)
/tmp/ipykernel_9110/4276621302.py in ()
6 .addAnalyzer(Mean("star_rating")) \
----> 8 .addAnalyzer(Histogram("star_rating", binningUdf= lambda x: x)) \
9 .run()
10
~/anaconda3/envs/python3/lib/python3.10/site-packages/pydeequ/analyzers.py in addAnalyzer(self, analyzer)
132 """
133 analyzer._set_jvm(self._jvm)
--> 134 _analyzer_jvm = analyzer._analyzer_jvm
135 self._AnalysisRunBuilder.addAnalyzer(_analyzer_jvm)
136 return self
~/anaconda3/envs/python3/lib/python3.10/site-packages/pydeequ/analyzers.py in _analyzer_jvm(self)
460 return self._deequAnalyzers.Histogram(
461 self.column,
--> 462 self._jvm.scala.Option.apply(self.binningUdf),
463 self.maxDetailBins,
464 self._jvm.scala.Option.apply(self.where),
~/anaconda3/envs/python3/lib/python3.10/site-packages/py4j/java_gateway.py in __call__(self, *args)
1311
1312 def __call__(self, *args):
-> 1313 args_command, temp_args = self._build_args(*args)
1314
1315 command = proto.CALL_COMMAND_NAME +\
~/anaconda3/envs/python3/lib/python3.10/site-packages/py4j/java_gateway.py in _build_args(self, *args)
1281
1282 args_command = "".join(
-> 1283 [get_command_part(arg, self.pool) for arg in new_args])
1284
1285 return args_command, temp_args
~/anaconda3/envs/python3/lib/python3.10/site-packages/py4j/java_gateway.py in (.0)
1281
1282 args_command = "".join(
-> 1283 [get_command_part(arg, self.pool) for arg in new_args])
1284
1285 return args_command, temp_args
~/anaconda3/envs/python3/lib/python3.10/site-packages/py4j/protocol.py in get_command_part(parameter, python_proxy_pool)
296 command_part += ";" + interface
297 else:
--> 298 command_part = REFERENCE_TYPE + parameter._get_object_id()
299
300 command_part += "\n"
AttributeError: 'function' object has no attribute '_get_object_id'
```
I have tried using a simple lambda function that actually does no binning but returns its input as output:
```
.addAnalyzer(Histogram("star_rating",binningUdf=lambda x: x))
```
**To Reproduce**
Steps to reproduce the behavior:
1. Go to the [tutorial on analyzers](https://github.com/awslabs/python-deequ/blob/master/tutorials/analyzers.ipynb)
3. Scroll down to command 3 and add `.addAnalyzer(Histogram("star_rating", binningUdf=lambda x: x))`
**Expected behavior**
I would expect it to work just as it works when I'm doing it without binningUdf (i.e. just `.addAnalyzer(Histogram("star_rating"))`)
Some more documentation on how to use the binningUdf parameter would be great.
Contributor guide
Assessment
This issue has not been assessed yet.