awslabs / awslabs/python-deequ
Anomaly Check functions not defined when using PyDeequ on Glue
- Dominant language
- Jupyter Notebook
- Stars
- 826
- Forks
- 158
- Avg merge
- 9d 22h
- Merged PRs (30d)
- 3
Description
**Describe the bug**
While following this Tutorial:
https://github.com/awslabs/python-deequ/blob/master/tutorials/anomaly_detection.ipynb
Error:
```
File "", line 4, in
NameError: name 'RateOfChangeStrategy' is not defined
```
Same error for SimpleThresholdStrategy,RelativeRateOfChangeStrategy etc.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a Glue Jobs with same code as per the Tutorial : https://github.com/awslabs/python-deequ/blob/master/tutorials/anomaly_detection.ipynb
2. Except, before importing pydeequ create python environment variable :
```
import os
os.environ["SPARK_VERSION"] = "3.3"
```
3. Use Glue Version 4.0 . Spark 3.3
4. To include Pydeequ module create a setuup.py as shown below :
```
from setuptools import setup
setup(
name="pydeequ_module",
version="0.1",
packages=['pydeequ_module'],
install_requires=['pydeequ==1.1.0rc0','sagemaker_pyspark']
)
```
Copy the .whl file to s3 location and refer it as additional python libraries in Glue job. Reference :
https://repost.aws/knowledge-center/glue-import-error-no-module-named
5. Use the dependent jar file as deequ-2.0.3-spark-3.3.jar
6. Run the job the error shows up as :NameError: name 'RelativeRateOfChangeStrategy' is not defined
**Expected behavior**
The job should succeed identifying anomaly.
Contributor guide
Assessment
This issue has not been assessed yet.