Dimensions missing in druid for realtime data - They appear properly when data is moved to historical
- Dominant language
- Clojure
- Stars
- 49.3k
- Forks
- 6.8k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 653
Description
This is already explored in #2376, but was closed without a fix, and this is still an issue, but I think I can add more precisions on how to reproduce it.
I am using druid-0.10.1, but saw the same issue with druid versions prior to that (0.9, 0.10.0).
We are on `metabase/metabase-head:latest`.
This issue only happens when ingesting realtime metrics (via one of the realtime means, kafka-indexer, tranquility, etc...).
When ingesting realtime metrics (i.e, picking metrics using a realtime task: I am using the kafka-indexer for this right now, but the same happened when using tranquility stream server), metrics are shown right away, but druid's dimensions are not present in the table view, filters, group by, etc...
With the same data, if I use Batch ingestion (i.e, dump my data in a CSV file and use druid to batch ingest the file), dimensions and metrics are shown properly, and this issue _does not_ happens.
Moreover, my ingestion task is configured to move my realtime data to historical node every day. When this happens, the dimensions are then available for that data, just as when I was loading the same data via batch.
In short, if I wait one day, I can query my once-upon-a-time realtime data properly.
If I would have to bet, I would assume that the API to get the dimensions from druid for the realtime data is different than what we do for historical data.
I am happy to assist in anyway I can.
The other details
------
**Browsers**: Using Safari and Chrome interchangeably.
**OS**: Druid + metabase on Docker (alpine), Client on MacOSX
**Database**: Druid
**Metabase DB**: H2
Contributor guide
Research direction
Start by reproducing the issue with Druid realtime ingestion through the Kafka indexer or Tranquility, then compare dimension visibility before and after data moves to a historical node. Done means dimensions from realtime data appear in table views, filters, and group-by controls without waiting for historical handoff.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100