apache / apache/pinot

Ingestion spec templating is rather undocumented

Open
#5,883 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

While trying to set up ingestion, I ran across the `-propertyFile` and `-values` arguments in `LaunchDataIngestionJobCommand`. I didn't see anything in the docs about templating the job spec, and only later found one reference describing it in https://github.com/pinot-contrib/pinot-docs/blob/eb9a8a07687bfe78b022ba0825123fd43e316795/operators/cli.md.

This would be helpful to document and also answer questions such as:
* What the format of the `propertyFile` is.
* What the format for templated variables in the job spec file should be.

My particular use-case where this is great is in a setup where ingestion (via Spark or Hadoop) are only distributed as single JAR (compiled with deps, rather than distributed external JARs), and hooking up external file dependencies is a pain. For this, I'd ideally like to (1) bundle a basic configuration file as a resource in the JAR or as a separate distribution (2) provide any overrides at run-time via parameters (eg: by a scheduler application)

(Separately, it looks like using JAR resources for args like that aren't supported 🤔 I have to look a bit more into whether or not that's necessary. Is that normally a sane thing for Java applications to support?)

Contributor guide

Open the contributing guide

Research direction

Start with LaunchDataIngestionJobCommand and the existing reference in operators/cli.md from the linked pinot-docs revision. Determine and document the propertyFile format and templated-variable syntax for ingestion job specs, including the runtime override use case described here. Done means the relevant CLI documentation answers those format questions clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cli, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.