Add a Bash Script/Command for Spark Ingestion
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently, spark ingestion has lot of parameters that need to be changed based on environment. A command can be added to Pinot that takes the env (e.g. EMR, YARN, K8s), the pinot binary path, the plugin names and can generate a full spark command for you. This would help avoid confguring class paths, schems for the jar locations, java version and so one.
Contributor guide
Research direction
Start by mapping the current Spark ingestion parameters and how EMR, YARN, and K8s environments differ, including the Pinot binary path, plugin names, class paths, schemas, and Java version. The work is done when a Bash command accepts those inputs and generates a complete Spark ingestion command; the issue names no files or tests, so the relevant entry points and validation steps must first be identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100