Kafka Druid ingestion: only the first key field ingested and others ignored?
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
We’re trying to set up Kafka ingestion in Druid so that all fields from both the Kafka key and value are included in the datasource.
Right now, only the first field of the key is being ingested. I found this note in the documentation:
`
“The input format to parse the Kafka key only processes the first entry of the inputFormat field. If your key values are simple strings, you can use the tsv format to parse them. Note that for tsv, csv, and regex formats, you need to provide a columns array to make a valid input format. Only the first one is used, and its name will be ignored in favor of keyColumnName.”`
Did I miss something?
Is there a way to ingest all Kafka key fields as columns in the Druid datasource—without copying them into the value?
Contributor guide
Research direction
Start with the Kafka ingestion input-format documentation and the quoted key-parsing behavior; compare the supported formats and how keyColumnName is applied. Done means establishing whether multiple Kafka key fields can be ingested without copying them into the value, and documenting the limitation or supported path if they cannot.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100