awslabs / awslabs/python-deequ

Anomaly Check functions not defined when using PyDeequ on Glue

Open
#138 3 comments 0 reactions 1 assignee Claimed by @chenliu0831 View on GitHub
bug question
Dominant language
Jupyter Notebook
Stars
826
Forks
158
Avg merge
9d 22h
Merged PRs (30d)
3

Description

**Describe the bug**
While following this Tutorial:
https://github.com/awslabs/python-deequ/blob/master/tutorials/anomaly_detection.ipynb
Error:
```
File "", line 4, in
NameError: name 'RateOfChangeStrategy' is not defined
```
Same error for SimpleThresholdStrategy,RelativeRateOfChangeStrategy etc.

**To Reproduce**
Steps to reproduce the behavior:
1. Create a Glue Jobs with same code as per the Tutorial : https://github.com/awslabs/python-deequ/blob/master/tutorials/anomaly_detection.ipynb
2. Except, before importing pydeequ create python environment variable :
```
import os
os.environ["SPARK_VERSION"] = "3.3"
```

3. Use Glue Version 4.0 . Spark 3.3
4. To include Pydeequ module create a setuup.py as shown below :

```
from setuptools import setup

setup(
name="pydeequ_module",
version="0.1",
packages=['pydeequ_module'],
install_requires=['pydeequ==1.1.0rc0','sagemaker_pyspark']
)
```
Copy the .whl file to s3 location and refer it as additional python libraries in Glue job. Reference :
https://repost.aws/knowledge-center/glue-import-error-no-module-named
5. Use the dependent jar file as deequ-2.0.3-spark-3.3.jar
6. Run the job the error shows up as :NameError: name 'RelativeRateOfChangeStrategy' is not defined

**Expected behavior**
The job should succeed identifying anomaly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.