awslabs / awslabs/amazon-emr-cli

EMR CLI run throws error

Open
#24 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Hello I am using Mac OS running Docker container

```

emr run \
--entry-point entrypoint.py \
--application-id \
--job-role arn:aws:iam::043916019468:role/AmazonEMR-ExecutionRole-1693489747108 \
--s3-code-uri s3://jXXXv/emr_scripts/ \
--spark-submit-opts "--conf spark.jars=/usr/lib/hudi/hudi-spark-bundle.jar --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory" \
--build \
--wait
```

# Error
```
[emr-cli]: Packaging assets into dist/
[+] Building 23.2s (10/15) docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.76kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 83B 0.0s
=> [internal] load metadata for docker.io/library/amazonlinux:2 0.6s
=> [auth] library/amazonlinux:pull token for registry-1.docker.io 0.0s
=> [base 1/7] FROM docker.io/library/amazonlinux:2@sha256:e218c279c7954a94c6f4c8ab106c3ea389675d429feec903bc4c93fa66ed4fd0 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 486B 0.0s
=> CACHED [base 2/7] RUN yum install -y python3 tar gzip 0.0s
=> CACHED [base 3/7] RUN python3 -m venv /opt/venv 0.0s
=> CACHED [base 4/7] RUN python3 -m pip install --upgrade pip 0.0s
=> ERROR [base 5/7] RUN curl -sSL https://install.python-poetry.org | python3 - 22.6s
------
> [base 5/7] RUN curl -sSL https://install.python-poetry.org | python3 -:
22.53 Retrieving Poetry metadata
22.53
22.53 # Welcome to Poetry!
22.53
22.53 This will download and install the latest version of Poetry,
22.53 a dependency and package manager for Python.
22.53
22.53 It will add the `poetry` command to Poetry's bin directory, located at:
22.53
22.53 /root/.local/bin
22.53
22.53 You can uninstall at any time by executing this script with the --uninstall option,
22.53 and these changes will be reverted.
22.53
22.53 Installing Poetry (1.6.1)
22.53 Installing Poetry (1.6.1): Creating environment
22.53 Installing Poetry (1.6.1): Installing Poetry
22.53 Installing Poetry (1.6.1): An error occurred. Removing partial environment.
22.53 Poetry installation failed.
22.53 See /poetry-installer-error-ulqu13bo.log for error logs.
------
Dockerfile:32
--------------------
30 |
31 | RUN python3 -m pip install --upgrade pip
32 | >>> RUN curl -sSL https://install.python-poetry.org | python3 -
33 |
34 | ENV PATH="$PATH:/root/.local/bin"
--------------------
ERROR: failed to solve: process "/bin/sh -c curl -sSL https://install.python-poetry.org | python3 -" did not complete successfully: exit code: 1
Traceback (most recent call last):
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/bin/emr", line 8, in
sys.exit(cli())
^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/emr_cli/emr_cli.py", line 238, in run
p.build()
File "/Users/soumilnitinshah/IdeaProjects/DemoProject/venv/lib/python3.11/site-packages/emr_cli/packaging/python_project.py", line 56, in build

```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the reported `emr run --build --wait` command, then inspect the generated Dockerfile at line 32 and `emr_cli/packaging/python_project.py` around line 56. Confirm whether the packaging build completes successfully on the reported Amazon Linux and Docker setup, and document or verify the resulting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python
Domain
build-system, cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.