awslabs / awslabs/amazon-emr-cli
Add support for `--show-logs` in cluster mode on EMR on EC2
- Dominant language
- Python
- Stars
- 47
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
With the recent `--show-logs` flag, we switch the deploy mode to `client` so that EMR steps can capture the driver `stdout`.
Unfortunately, `--client` mode doesn't work with additional archives provided via the `--archives` flag or `--conf spark.archives` parameter. See https://issues.apache.org/jira/browse/SPARK-36088 for more a related issue.
In order to support this for cluster mode, we'd need to parse the step `stderr` logs to retrieve the Yarn application ID, then fetch the Yarn application logs from S3.
Contributor guide
Research direction
Trace the existing --show-logs handling and the cluster-mode path, then review how --archives and --conf spark.archives are passed to EMR. The implementation needs to parse step stderr for the YARN application ID and fetch the corresponding application logs from S3; done means --show-logs works in cluster mode with additional archives.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, spark
- Domain
- cli, cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100