apache / apache/pinot

Allow submiting ingestionJobSpec yaml to launch minion task for ingestion job

Open
#9,629 1 comment 0 reactions 0 assignees View on GitHub
ingestion user-experience
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Currently, users run this command to ingest data: `pinot-admin.sh LaunchIngestionJob -jobSpec my-ingestion-job-spec.yaml`.

The goal is to make users can re-use the same yaml for segment ingestion in minion.

Steps:
1. Add an endpoint in the Pinot controller to allow users to POST the yaml file
2. read yaml to SegmentGenerationJobSpec, launch minion tasks
3. Return the tasks generated map.

For simplicity, if this minion task can honor all the configs from SegmentGenerationJobSpec, then we can just convert SegmentGenerationJobSpec to a SegmentGenerationAndPushTask Minion task.
And if not, we need to wrap those `SegmentGenerationJobRunner` and `SegmentMetadataPushJobRunner` jobs and make them minon tasks.

Ref:
org.apache.pinot.core.query.executor.sql.SqlQueryExecutor : how Insert INTO statement is executed.
org.apache.pinot.plugin.minion.tasks.segmentgenerationandpush.SegmentGenerationAndPushTaskGenerator: how the underlying minion task generator and executor.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.