Allow submiting ingestionJobSpec yaml to launch minion task for ingestion job
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Currently, users run this command to ingest data: `pinot-admin.sh LaunchIngestionJob -jobSpec my-ingestion-job-spec.yaml`.
The goal is to make users can re-use the same yaml for segment ingestion in minion.
Steps:
1. Add an endpoint in the Pinot controller to allow users to POST the yaml file
2. read yaml to SegmentGenerationJobSpec, launch minion tasks
3. Return the tasks generated map.
For simplicity, if this minion task can honor all the configs from SegmentGenerationJobSpec, then we can just convert SegmentGenerationJobSpec to a SegmentGenerationAndPushTask Minion task.
And if not, we need to wrap those `SegmentGenerationJobRunner` and `SegmentMetadataPushJobRunner` jobs and make them minon tasks.
Ref:
org.apache.pinot.core.query.executor.sql.SqlQueryExecutor : how Insert INTO statement is executed.
org.apache.pinot.plugin.minion.tasks.segmentgenerationandpush.SegmentGenerationAndPushTaskGenerator: how the underlying minion task generator and executor.
Contributor guide
Assessment
This issue has not been assessed yet.