apache / apache/druid

[PROPOSAL] Support only finiteFirehose for native batch ingestion

Open
#7,071 4 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Incompatible Proposal stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

# Motivation

Currently native batch tasks (local and parallel index tasks) support any firehose implementation. However, it isn't very useful when firehose is an infinite one because they don't have any context about stream ingestion.

# Proposed changes

I propose to change the type of `firehose` of `IndexIOConfig` and `ParallelIndexIOConfig` from `FirehoseFactory` to `FiniteFirehoseFactory`.

# Rationale

`FiniteFirehoseFactory` is designed for any type of batch ingestion. It assumes that input data is finite (and provides an optional hint for parallel indexing). It makes more sense to support only `FiniteFirehoseFactory` for native batch tasks rather than improve them to support any kind of firehoseFactory which may be designed for stream input data.

# Operational impact

There's no change in the task spec because the variable name isn't changed.

Custom firehoseFactory implementations for native batch tasks need to be updated.

# Future work

This change effectively makes native batch tasks to support only text file formats by default because all implementations of `FiniteFirehoseFactory` are using `StringInputRowParser`. https://github.com/apache/incubator-druid/issues/5584 should be solved to support various file formats.

Contributor guide

Open the contributing guide

Research direction

Locate IndexIOConfig and ParallelIndexIOConfig and inspect their firehose fields and usages; then read FiniteFirehoseFactory and the native batch task wiring. Check affected task specs and custom factory implementations for compatibility, and confirm native batch ingestion accepts only finite factories without changing the task-spec field name.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.