[PROPOSAL] Support only finiteFirehose for native batch ingestion
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
# Motivation
Currently native batch tasks (local and parallel index tasks) support any firehose implementation. However, it isn't very useful when firehose is an infinite one because they don't have any context about stream ingestion.
# Proposed changes
I propose to change the type of `firehose` of `IndexIOConfig` and `ParallelIndexIOConfig` from `FirehoseFactory` to `FiniteFirehoseFactory`.
# Rationale
`FiniteFirehoseFactory` is designed for any type of batch ingestion. It assumes that input data is finite (and provides an optional hint for parallel indexing). It makes more sense to support only `FiniteFirehoseFactory` for native batch tasks rather than improve them to support any kind of firehoseFactory which may be designed for stream input data.
# Operational impact
There's no change in the task spec because the variable name isn't changed.
Custom firehoseFactory implementations for native batch tasks need to be updated.
# Future work
This change effectively makes native batch tasks to support only text file formats by default because all implementations of `FiniteFirehoseFactory` are using `StringInputRowParser`. https://github.com/apache/incubator-druid/issues/5584 should be solved to support various file formats.
Contributor guide
Research direction
Locate IndexIOConfig and ParallelIndexIOConfig and inspect their firehose fields and usages; then read FiniteFirehoseFactory and the native batch task wiring. Check affected task specs and custom factory implementations for compatibility, and confirm native batch ingestion accepts only finite factories without changing the task-spec field name.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100