Don't store all-null dimension columns
Open
Area - Batch Ingestion
Performance
stale
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 31m
- Merged PRs (30d)
- 209
Description
When a Druid segment is created, it could be checked that a dictionary-encoded dimension column contains all null values, and just not written to the segment if so.
Probably the same couldn't be done for all-zero numeric columns, because if they are queried as dimensions they should appear as dimensions with all "0" values, not "null" values.
Contributor guide
Research direction
No files, tests, or entry points are named in the issue. Start by tracing Druid segment creation and dictionary-encoded dimension-column writing, then check how omitted all-null columns behave when queried. Done means all-null dimension columns are not written while their query semantics remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100