apache / apache/pinot

Add a Bash Script/Command for Spark Ingestion

Open
#9,009 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, spark ingestion has lot of parameters that need to be changed based on environment. A command can be added to Pinot that takes the env (e.g. EMR, YARN, K8s), the pinot binary path, the plugin names and can generate a full spark command for you. This would help avoid confguring class paths, schems for the jar locations, java version and so one.

Contributor guide

Open the contributing guide

Research direction

Start by mapping the current Spark ingestion parameters and how EMR, YARN, and K8s environments differ, including the Pinot binary path, plugin names, class paths, schemas, and Java version. The work is done when a Bash command accepts those inputs and generates a complete Spark ingestion command; the issue names no files or tests, so the relevant entry points and validation steps must first be identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.