awslabs / awslabs/python-deequ
Anomaly Check functions not defined when using PyDeequ on Glue
@chenliu0831 arbeitet bereits daran.
Seit 21.8.2023.
- Vorherrschende Sprache
- Jupyter Notebook
- Sterne
- 826
- Forks
- 158
- Ø Merge
- 9 T. 22 Std.
- Gemergte PRs (30 T.)
- 3
Beschreibung
Describe the bug
While following this Tutorial:
https://github.com/awslabs/python-deequ/blob/master/tutorials/anomaly_detection.ipynb
Error:
File "<stdin>", line 4, in <module>
NameError: name 'RateOfChangeStrategy' is not defined
Same error for SimpleThresholdStrategy,RelativeRateOfChangeStrategy etc.
To Reproduce
Steps to reproduce the behavior:
- Create a Glue Jobs with same code as per the Tutorial : https://github.com/awslabs/python-deequ/blob/master/tutorials/anomaly_detection.ipynb
- Except, before importing pydeequ create python environment variable :
import os
os.environ["SPARK_VERSION"] = "3.3"
- Use Glue Version 4.0 . Spark 3.3
- To include Pydeequ module create a setuup.py as shown below :
from setuptools import setup
setup(
name="pydeequ_module",
version="0.1",
packages=['pydeequ_module'],
install_requires=['pydeequ==1.1.0rc0','sagemaker_pyspark']
)
Copy the .whl file to s3 location and refer it as additional python libraries in Glue job. Reference :
https://repost.aws/knowledge-center/glue-import-error-no-module-named
5. Use the dependent jar file as deequ-2.0.3-spark-3.3.jar
6. Run the job the error shows up as :NameError: name 'RelativeRateOfChangeStrategy' is not defined
Expected behavior
The job should succeed identifying anomaly.
Beitragsleitfaden
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Bewertung
Dieses Issue wurde noch nicht bewertet.