dask / dask/dask-yarn

AWS EMR bootstrap script fails

Open
#133 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
69
Forks
40
PR merge metrics
No merged PRs in 30d

Description

**What happened**:
I tried to instantiate an EMR cluster with the AWS EMR bootstrap script. The provisioning of the cluster failed saying "bootstrap action 1 returned a non-zero return code". Extracting the logs, the stdout of the master arrives to call the conda pack command:
```
...
Installing base packages
Packaging environment

```
while the stderr shows this:
```
...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:

- python='3.8.5'

This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).
```

**What you expected to happen**:
The EMR cluster provisions correctly.

**Minimal Complete Verifiable Example**:
I first uploaded the bootstrap script to s3://my-bucket/dask-emr/bootstrap.sh than tried to provision the EMR cluster with this AWS CLI command:
```sh
aws emr create-cluster --name "Janos Dask Trial Cluster" \
--release-label emr-5.30.1 \
--applications Name=HADOOP \
--log-uri s3://my-bucket/dask-emr/logs/ \
--ec2-attributes KeyName=my-key \
--instance-type m5.xlarge \
--instance-count 3 \
--bootstrap-actions Path=s3://my-bucket/dask-emr/bootstrap.sh,Args="[--conda-packages,bokeh,fastparquet,python-snappy,snappy]" \
--use-default-roles
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.