awslabs / awslabs/amazon-emr-cli
EMR CLI run throws error
- Dominant language
- Python
- Stars
- 47
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
Hello I am using Mac OS running Docker container
```
emr run \
--entry-point entrypoint.py \
--application-id \
--job-role arn:aws:iam::043916019468:role/AmazonEMR-ExecutionRole-1693489747108 \
--s3-code-uri s3://jXXXv/emr_scripts/ \
--spark-submit-opts "--conf spark.jars=/usr/lib/hudi/hudi-spark-bundle.jar --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory" \
--build \
--wait
```
# Error
```
[emr-cli]: Packaging assets into dist/
[+] Building 23.2s (10/15) docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.76kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 83B 0.0s
=> [internal] load metadata for docker.io/library/amazonlinux:2 0.6s
=> [auth] library/amazonlinux:pull token for registry-1.docker.io 0.0s
=> [base 1/7] FROM docker.io/library/amazonlinux:2@sha256:e218c279c7954a94c6f4c8ab106c3ea389675d429feec903bc4c93fa66ed4fd0 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 486B 0.0s
=> CACHED [base 2/7] RUN yum install -y python3 tar gzip 0.0s
=> CACHED [base 3/7] RUN python3 -m venv /opt/venv 0.0s
=> CACHED [base 4/7] RUN python3 -m pip install --upgrade pip 0.0s
=> ERROR [base 5/7] RUN curl -sSL https://install.python-poetry.org | python3 - 22.6s
------
> [base 5/7] RUN curl -sSL https://install.python-poetry.org | python3 -:
22.53 Retrieving Poetry metadata
22.53
22.53 # Welcome to Poetry!
22.53
22.53 This will download and install the latest version of Poetry,
22.53 a dependency and package manager for Python.
22.53
22.53 It will add the `poetry` command to Poetry's bin directory, located at:
22.53
22.53 /root/.local/bin
22.53
22.53 You can uninstall at any time by executing this script with the --uninstall option,
22.53 and these changes will be reverted.
22.53
22.53 Installing Poetry (1.6.1)
22.53 Installing Poetry (1.6.1): Creating environment
22.53 Installing Poetry (1.6.1): Installing Poetry
22.53 Installing Poetry (1.6.1): An error occurred. Removing partial environment.
22.53 Poetry installation failed.
22.53 See /poetry-installer-error-ulqu13bo.log for error logs.
------
Dockerfile:32
--------------------
30 |
31 | RUN python3 -m pip install --upgrade pip
32 | >>> RUN curl -sSL https://install.python-poetry.org | python3 -
33 |
34 | ENV PATH="$PATH:/root/.local/bin"
--------------------
ERROR: failed to solve: process "/bin/sh -c curl -sSL https://install.python-poetry.org | python3 -" did not complete successfully: exit code: 1
Traceback (most recent call last):
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/bin/emr", line 8, in
sys.exit(cli())
^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/emr_cli/emr_cli.py", line 238, in run
p.build()
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/emr_cli/packaging/python_project.py", line 56, in build
```
Contributor guide
Research direction
Reproduce the failure with the reported `emr run --build --wait` command, then inspect the generated Dockerfile at line 32 and `emr_cli/packaging/python_project.py` around line 56. Confirm whether the packaging build completes successfully on the reported Amazon Linux and Docker setup, and document or verify the resulting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- build-system, cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100