apache / apache/druid

Kafka Druid ingestion: only the first key field ingested and others ignored?

Open
#18,578 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

We’re trying to set up Kafka ingestion in Druid so that all fields from both the Kafka key and value are included in the datasource.

Right now, only the first field of the key is being ingested. I found this note in the documentation:
`
“The input format to parse the Kafka key only processes the first entry of the inputFormat field. If your key values are simple strings, you can use the tsv format to parse them. Note that for tsv, csv, and regex formats, you need to provide a columns array to make a valid input format. Only the first one is used, and its name will be ignored in favor of keyColumnName.”`

Did I miss something?
Is there a way to ingest all Kafka key fields as columns in the Druid datasource—without copying them into the value?

Contributor guide

Open the contributing guide

Research direction

Start with the Kafka ingestion input-format documentation and the quoted key-parsing behavior; compare the supported formats and how keyColumnName is applied. Done means establishing whether multiple Kafka key fields can be ingested without copying them into the value, and documenting the limitation or supported path if they cannot.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.