(aws-glue): Glue Job import failure when using extra python files
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
Glue Python job imports fail when the jobs are created using `glue.Job` and make use of `extra_python_files` and `glue.Code.from_asset`. CDK seems to be renaming the imported file to a hash value.
```
job_with_extra_executable = glue.JobExecutable.python_etl(
glue_version=glue.GlueVersion.V2_0,
python_version=glue.PythonVersion.THREE,
script=glue.Code.from_asset("./example_with_extra.py"),
extra_python_files=[glue.Code.from_asset("./custom_functions.py")],
)
job_with_extra = glue.Job(
self, "JobWithExtra",
job_name="job_with_extra",
executable=job_with_extra_executable,
worker_type=glue.WorkerType.G_2_X,
worker_count=3
)
```
### Expected Behavior
Import succeeds in Pyspark job.
### Current Behavior
Import fails in Pyspark job.
### Reproduction Steps
app.py
```
import aws_cdk as cdk
from stacks import GlueStack
app = cdk.App()
GlueStack(app, "GlueStack")
app.synth()
```
stacks.py
```
from aws_cdk import Stack
import aws_cdk.aws_glue_alpha as glue
from constructs import Construct
class GlueStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs):
super().__init__(scope, construct_id, **kwargs)
job_with_extra_executable = glue.JobExecutable.python_etl(
glue_version=glue.GlueVersion.V2_0,
python_version=glue.PythonVersion.THREE,
script=glue.Code.from_asset("./example_with_extra.py"),
extra_python_files=[glue.Code.from_asset("./custom_functions.py")],
)
job_with_extra = glue.Job(
self, "JobWithExtra",
job_name="job_with_extra",
executable=job_with_extra_executable,
worker_type=glue.WorkerType.G_2_X,
worker_count=3
)
```
example_with_extra.py
```
import sys
import os
from pyspark.sql import SparkSession
from awsglue.utils import getResolvedOptions
def dump_python_path():
print("python path:", sys.path)
for path in sys.path:
if os.path.isdir(path):
print(f"dir: {path}")
print("\t" + str(os.listdir(path)))
print(path)
def main():
want_args = ["JOB_NAME",]
args = getResolvedOptions(sys.argv, want_args)
spark = (
SparkSession
.builder
.getOrCreate()
)
from custom_functions import make_rdd
rdd = make_rdd(spark.sparkContext, 10)
print(rdd.collect())
if __name__ in ["__main__", "builtins"]:
dump_python_path()
main()
```
custom_functions.py
```
def make_rdd(sc, n: int):
return sc.parallelize(range(1, n))
```
### Possible Solution
It appears that CDK/Cfn is renaming the Python module file to a random hash value when it copies them to S3 and then into the Glue Spark cluster causing the import to fail. The following is the output when running the deployed `job_with_extra` Glue job:
```
python path: ['/opt/amazon/bin', '/tmp/localPyFiles-b93e2816-a167-4622-9a47-2209331824ec', '/opt/amazon/spark/jars/spark-core_2.11-2.4.3.jar', '/opt/amazon/spark/python/lib/pyspark.zip', '/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip', '/opt/amazon/lib/python3.6/site-packages', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/spark/.local/lib/python3.7/site-packages', '/usr/local/lib64/python3.7/site-packages', '/usr/local/lib/python3.7/site-packages', '/usr/lib64/python3.7/site-packages', '/usr/lib/python3.7/site-packages']
dir: /opt/amazon/bin
['entry.sh', 'runscript.py']
/opt/amazon/bin
dir: /tmp/localPyFiles-b93e2816-a167-4622-9a47-2209331824ec
['92d0ce32e33acc96de36bc082a7e6fb40eb9bf01acdbd0774e0152c3424a839e.py']
/tmp/localPyFiles-b93e2816-a167-4622-9a47-2209331824ec
/opt/amazon/spark/jars/spark-core_2.11-2.4.3.jar
/opt/amazon/spark/python/lib/pyspark.zip
/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip
dir: /opt/amazon/lib/python3.6/site-packages
['Spark-1.0-py3.6.egg-info', 'pyspark', 'AwsGlueMlMarconiWrapperPython-1.0-py3.6.egg-info', 'AWSGlueDataplanePython-1.0-py3.6.egg-info', 'awsglue', 'awsglueml', 'aws_glue_custom_connector_python', 'AWSGlueCustomConnectorPython-1.0-py3.6.egg-info']
/opt/amazon/lib/python3.6/site-packages
/usr/lib64/python37.zip
dir: /usr/lib64/python3.7
['ast.py', 'concurrent', 'sysconfig.py', 'operator.py', 'ipaddress.py', 'calendar.py', 'asyncore.py', 'threading.py', 'ssl.py', 'optparse.py', 'fractions.py', 'config-3.7m-x86_64-linux-gnu', 'linecache.py', 'trace.py', '_threading_local.py', 'quopri.py', 'code.py', 'encodings', 'sqlite3', 'struct.py', 'pty.py', 'sre_parse.py', 'pydoc.py', 'mailcap.py', 'py_compile.py', 'cmd.py', 'genericpath.py', 'netrc.py', 'base64.py', 'platform.py', 'functools.py', '_sitebuiltins.py', 'lib2to3', 'copy.py', 'reprlib.py', 'keyword.py', 'sndhdr.py', 'mimetypes.py', 'pyclbr.py', 'enum.py', 'pickle.py', 'abc.py', 'tty.py', 'chunk.py', 'profile.py', 'pathlib.py', 'symbol.py', 'glob.py', 'distutils', 'smtpd.py', 'site.py', 'queue.py', 'imp.py', 'nturl2path.py', 'uu.py', 'contextvars.py', 'compileall.py', 'imaplib.py', 'this.py', 'datetime.py', 'difflib.py', 'gettext.py', 'stat.py', 'io.py', 'argparse.py', 'zipapp.py', 'bdb.py', 'pydoc_data', 'asynchat.py', 'types.py', 'getpass.py', 'asyncio', 'antigravity.py', 'codeop.py', 'json', 'doctest.py', '_sysconfigdata_dm_linux_x86_64-linux-gnu.py', '_bootlocale.py', 'crypt.py', 'macpath.py', 'curses', 'email', '_collections_abc.py', '_sysconfigdata_m_linux_x86_64-linux-gnu.py', 'ftplib.py', 'string.py', 'aifc.py', 'importlib', 'shelve.py', 'sre_constants.py', '__pycache__', 'dis.py', 'cProfile.py', 'pickletools.py', 'token.py', 'venv', 'socketserver.py', 'pstats.py', 'contextlib.py', 'weakref.py', 'tokenize.py', 'binhex.py', 'gzip.py', 'tracemalloc.py', 'signal.py', 'smtplib.py', 'selectors.py', '_pydecimal.py', 'site-packages', 'wsgiref', '_osx_support.py', 'poplib.py', '_dummy_thread.py', 'decimal.py', 'logging', 'fileinput.py', 'runpy.py', '_markupbase.py', 'urllib', 'posixpath.py', 'pipes.py', '_compression.py', 'configparser.py', 'locale.py', 'cgitb.py', 're.py', 'test', 'csv.py', '_compat_pickle.py', 'lzma.py', 'random.py', '_strptime.py', 'formatter.py', 'copyreg.py', 'codecs.py', 'lib-dynload', 'dbm', 'textwrap.py', 'colorsys.py', 'statistics.py', 'os.py', 'warnings.py', 'typing.py', 'timeit.py', 'tarfile.py', 'nntplib.py', 'hashlib.py', 'heapq.py', 'uuid.py', 'getopt.py', 'shutil.py', 'multiprocessing', 'pdb.py', 'traceback.py', '_pyio.py', 'wave.py', 'numbers.py', 'hmac.py', 'zipfile.py', 'socket.py', 'plistlib.py', 'cgi.py', 'pkgutil.py', 'http', 'ensurepip', 'rlcompleter.py', 'fnmatch.py', 'unittest', 'webbrowser.py', 'bisect.py', 'secrets.py', 'sched.py', '_py_abc.py', 'ntpath.py', 'dummy_threading.py', 'pprint.py', 'telnetlib.py', 'imghdr.py', 'shlex.py', 'bz2.py', 'tabnanny.py', 'ctypes', 'stringprep.py', 'html', 'sre_compile.py', 'xdrlib.py', 'collections', 'inspect.py', 'modulefinder.py', '_weakrefset.py', 'mailbox.py', '__future__.py', 'dataclasses.py', '__phello__.foo.py', 'subprocess.py', 'xmlrpc', 'opcode.py', 'xml', 'symtable.py', 'sunau.py', 'tempfile.py', 'filecmp.py']
/usr/lib64/python3.7
dir: /usr/lib64/python3.7/lib-dynload
['cmath.cpython-37m-x86_64-linux-gnu.so', '_ctypes.cpython-37m-x86_64-linux-gnu.so', '_lsprof.cpython-37m-x86_64-linux-gnu.so', '_asyncio.cpython-37m-x86_64-linux-gnu.so', 'array.cpython-37m-x86_64-linux-gnu.so', '_struct.cpython-37m-x86_64-linux-gnu.so', '_lzma.cpython-37m-x86_64-linux-gnu.so', 'select.cpython-37m-x86_64-linux-gnu.so', '_codecs_hk.cpython-37m-x86_64-linux-gnu.so', '_crypt.cpython-37m-x86_64-linux-gnu.so', 'termios.cpython-37m-x86_64-linux-gnu.so', 'xxlimited.cpython-37m-x86_64-linux-gnu.so', 'binascii.cpython-37m-x86_64-linux-gnu.so', '_codecs_kr.cpython-37m-x86_64-linux-gnu.so', '_sqlite3.cpython-37m-x86_64-linux-gnu.so', '_opcode.cpython-37m-x86_64-linux-gnu.so', '_ssl.cpython-37m-x86_64-linux-gnu.so', '_uuid.cpython-37m-x86_64-linux-gnu.so', 'parser.cpython-37m-x86_64-linux-gnu.so', 'fcntl.cpython-37m-x86_64-linux-gnu.so', 'pyexpat.cpython-37m-x86_64-linux-gnu.so', '_bz2.cpython-37m-x86_64-linux-gnu.so', 'nis.cpython-37m-x86_64-linux-gnu.so', '_sha256.cpython-37m-x86_64-linux-gnu.so', '_pickle.cpython-37m-x86_64-linux-gnu.so', 'mmap.cpython-37m-x86_64-linux-gnu.so', '_csv.cpython-37m-x86_64-linux-gnu.so', 'resource.cpython-37m-x86_64-linux-gnu.so', '_sha3.cpython-37m-x86_64-linux-gnu.so', '_datetime.cpython-37m-x86_64-linux-gnu.so', '_hashlib.cpython-37m-x86_64-linux-gnu.so', 'audioop.cpython-37m-x86_64-linux-gnu.so', '_json.cpython-37m-x86_64-linux-gnu.so', 'syslog.cpython-37m-x86_64-linux-gnu.so', '_codecs_jp.cpython-37m-x86_64-linux-gnu.so', 'unicodedata.cpython-37m-x86_64-linux-gnu.so', '_elementtree.cpython-37m-x86_64-linux-gnu.so', '_curses_panel.cpython-37m-x86_64-linux-gnu.so', '_curses.cpython-37m-x86_64-linux-gnu.so', '_queue.cpython-37m-x86_64-linux-gnu.so', '_dbm.cpython-37m-x86_64-linux-gnu.so', '_gdbm.cpython-37m-x86_64-linux-gnu.so', '_bisect.cpython-37m-x86_64-linux-gnu.so', '_heapq.cpython-37m-x86_64-linux-gnu.so', 'grp.cpython-37m-x86_64-linux-gnu.so', '_contextvars.cpython-37m-x86_64-linux-gnu.so', '_testmultiphase.cpython-37m-x86_64-linux-gnu.so', '_decimal.cpython-37m-x86_64-linux-gnu.so', 'ossaudiodev.cpython-37m-x86_64-linux-gnu.so', '_multibytecodec.cpython-37m-x86_64-linux-gnu.so', '_multiprocessing.cpython-37m-x86_64-linux-gnu.so', 'readline.cpython-37m-x86_64-linux-gnu.so', '_codecs_cn.cpython-37m-x86_64-linux-gnu.so', '_codecs_tw.cpython-37m-x86_64-linux-gnu.so', 'zlib.cpython-37m-x86_64-linux-gnu.so', 'math.cpython-37m-x86_64-linux-gnu.so', 'spwd.cpython-37m-x86_64-linux-gnu.so', '_blake2.cpython-37m-x86_64-linux-gnu.so', '_random.cpython-37m-x86_64-linux-gnu.so', '_posixsubprocess.cpython-37m-x86_64-linux-gnu.so', '_sha1.cpython-37m-x86_64-linux-gnu.so', '_codecs_iso2022.cpython-37m-x86_64-linux-gnu.so', '_socket.cpython-37m-x86_64-linux-gnu.so', '_sha512.cpython-37m-x86_64-linux-gnu.so', '_md5.cpython-37m-x86_64-linux-gnu.so']
/usr/lib64/python3.7/lib-dynload
dir: /home/spark/.local/lib/python3.7/site-packages
['pymysql', 'nltk-3.5-py3.7.egg-info', '_pydevd_bundle', 'mpl_toolkits', 'requests', 'pydevd_concurrency_analyser', 'certifi', 'python_dateutil-2.8.1.dist-info', 'scikit_learn-0.22.1.dist-info', 'matplotlib-3.1.3-py3.7-nspkg.pth', 'scipy', 'tbats-1.0.9.dist-info', 'zipp.py', 'jmespath-0.9.4.dist-info', 'requests-2.23.0.dist-info', 'Cython', 'mpmath', 'statsmodels', 'PyYAML-5.3.1-py3.7.egg-info', 'pytz', 'cycler.py', 'dateutil', 'enum34-1.1.9.dist-info', 'click-8.1.2.dist-info', 'urllib3-1.25.8.dist-info', 'pyarrow', 'regex-2022.3.15.dist-info', 'fsspec', 'fsspec-0.6.2.dist-info', 'pydev_app_engine_debug_startup.py', 'pylab.py', 'pyparsing-2.4.6.dist-info', 'sympy', 'pydev_ipython', 'easy_install.py', 'pmdarima', 'pydevd.py', 'chardet-3.0.4.dist-info', 'kiwisolver-1.1.0.dist-info', 'cycler-0.10.0.dist-info', '_pydev_runfiles', 'regex', 'wheel', 'botocore-1.15.4.dist-info', '_pydevd_frame_eval', 'pydevd_file_utils.py', 'pydevd_tracing.py', 'pydevd_plugins', 'pyhocon-0.3.54.dist-info', 'idna', 'typing_extensions.py', '__pycache__', 'boto3-1.12.4.dist-info', 'typing_extensions-4.1.1.dist-info', 's3fs-0.4.0.dist-info', 'tbats', 'six.py', 'tqdm', 'matplotlib', 'pyhocon', 'ptvsd-4.3.2.dist-info', 'pandas-1.0.1.dist-info', 'certifi-2019.11.28.dist-info', 'setuptools-45.2.0.dist-info', 'yaml', 'tqdm-4.64.0.dist-info', 'subprocess32-3.5.4-py3.7.egg-info', 'joblib-0.14.1.dist-info', 'setuptools', 'ptvsd', 'boto3', 'six-1.14.0.dist-info', 'pkg_resources', 'numpy-1.18.1.dist-info', 'cython.py', 'wheel-0.35.1.dist-info', 'zipp-3.8.0.dist-info', 's3fs', 'pyparsing.py', 'mpmath-1.1.0-py3.7.egg-info', 's3transfer', 'botocore', 'kiwisolver.cpython-37m-x86_64-linux-gnu.so', 'Cython-0.29.15.dist-info', 'chardet', 'pandas', 'setup_cython.py', 'click', 'idna-2.9.dist-info', 'enum', 'numpy', 'jmespath', 'avro_python3-1.10.0.dist-info', '_pydev_imps', 'patsy', 'pydevconsole.py', 'urllib3', 'pydevd_attach_to_process', 'importlib_metadata', 'PyMySQL-0.9.3.dist-info', 'pmdarima-1.5.3.dist-info', 'nltk', 'pyximport', 'scipy-1.4.1.dist-info', 'joblib', '_pydev_bundle', 'patsy-0.5.1.dist-info', 'matplotlib-3.1.3.dist-info', 'pytz-2019.3.dist-info', 's3transfer-0.3.3.dist-info', 'isympy.py', 'pydevd-1.9.0-py3.7.egg-info', 'avro', 'subprocess32', 'sympy-1.5.1.dist-info', 'statsmodels-0.11.1.dist-info', 'pyarrow-0.16.0.dist-info', 'sklearn', 'pydev_run_in_console.py', 'importlib_metadata-4.11.3.dist-info']
/home/spark/.local/lib/python3.7/site-packages
dir: /usr/local/lib64/python3.7/site-packages
['PyYAML-5.4.1.dist-info', 'yaml', '_yaml']
/usr/local/lib64/python3.7/site-packages
dir: /usr/local/lib/python3.7/site-packages
['colorama-0.4.3.dist-info', 'pyasn1', 'pyasn1-0.4.8.dist-info', 'dateutil', 'awscli', 'docutils', '__pycache__', 'awscli-1.22.94.dist-info', 'six.py', 'docutils-0.15.2.dist-info', 'six-1.16.0.dist-info', 'rsa', 's3transfer', 'botocore-1.24.39.dist-info', 'python_dateutil-2.8.2.dist-info', 'botocore', 'jmespath', 's3transfer-0.5.2.dist-info', 'urllib3', 'colorama', 'jmespath-1.0.0.dist-info', 'urllib3-1.26.9.dist-info', 'rsa-4.7.2.dist-info']
/usr/local/lib/python3.7/site-packages
dir: /usr/lib64/python3.7/site-packages
['__pycache__', 'README.txt']
/usr/lib64/python3.7/site-packages
dir: /usr/lib/python3.7/site-packages
['pip-20.2.2.dist-info', 'easy_install.py', '__pycache__', 'setuptools-49.1.3.dist-info', 'setuptools', 'pkg_resources', 'pip']
/usr/lib/python3.7/site-packages
2022-05-24 20:48:31,559 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Error from Python:Traceback (most recent call last):
File "/tmp/2b7104c33ad8577e958bd8053045e8a5b6ba5ab90bbebd9ea14f0c47af559a26.py", line 36, in
main()
File "/tmp/2b7104c33ad8577e958bd8053045e8a5b6ba5ab90bbebd9ea14f0c47af559a26.py", line 28, in main
from custom_functions import make_rdd
ModuleNotFoundError: No module named 'custom_functions'
```
In this case it looks like the Python file supplied to extra_python_files was renamed to: `92d0ce32e33acc96de36bc082a7e6fb40eb9bf01acdbd0774e0152c3424a839e.py`
```
aws s3 --quiet cp s3://cdk-XXXXXX-assets-XXXXXXXXXXXX-us-east-1/92d0ce32e33acc96de36bc082a7e6fb40eb9bf01acdbd0774e0152c3424a839e.py /dev/stdout
def make_rdd(sc, n: int):
return sc.parallelize(range(1, n))
```
### Additional Information/Context
_No response_
### CDK CLI Version
2.3.0 (build beaa5b2)
### Framework Version
_No response_
### Node.js Version
6.14.15
### OS
MacOS
### Language
Python
### Language Version
Python 3.8.9
### Other information
_No response_
Contributor guide
Research direction
Start with the GlueStack example in stacks.py and the aws_cdk.aws_glue_alpha Job and Code.from_asset APIs. Reproduce the deployment using app.py, example_with_extra.py, and custom_functions.py, then inspect the generated asset names and Glue job configuration. Done means the deployed PySpark job can import custom_functions from extra_python_files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100