Review and simplify EMR spark configuration in deployment scripts
- Dominant language
- Scala
- Stars
- 81
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
We've cargo-culted around a number of configuration options that make our EMR spark conf way more complicated than it needs to be. This task is to reduce the json configuration for the `batch-process.sh` script at `deployment/streaming/scripts/emr-configurations/batch-process.json` to a minimum set of options required to successfully complete that task.
Once reduced, we can begin to review other configurations for completeness.
Original discussion: https://github.com/azavea/osmesa/pull/184#discussion_r367622476
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading deployment/streaming/scripts/emr-configurations/batch-process.json and the linked discussion in pull request 184 to understand why each option was added. Run the batch-process.sh task with the reduced configuration and confirm it still completes successfully; done means only the required Spark and EMR settings remain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, spark
- Domain
- cloud, data-engineering
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- Half a day
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100