metabase / metabase/metabase

Dimensions missing in druid for realtime data - They appear properly when data is moved to historical

Open
#5,871 2 comments 2 reactions 0 assignees View on GitHub
Database/Druid Type:New Feature
Dominant language
Clojure
Stars
49.3k
Forks
6.8k
Avg merge
1d 13h
Merged PRs (30d)
653

Description

This is already explored in #2376, but was closed without a fix, and this is still an issue, but I think I can add more precisions on how to reproduce it.

I am using druid-0.10.1, but saw the same issue with druid versions prior to that (0.9, 0.10.0).
We are on `metabase/metabase-head:latest`.

This issue only happens when ingesting realtime metrics (via one of the realtime means, kafka-indexer, tranquility, etc...).

When ingesting realtime metrics (i.e, picking metrics using a realtime task: I am using the kafka-indexer for this right now, but the same happened when using tranquility stream server), metrics are shown right away, but druid's dimensions are not present in the table view, filters, group by, etc...

With the same data, if I use Batch ingestion (i.e, dump my data in a CSV file and use druid to batch ingest the file), dimensions and metrics are shown properly, and this issue _does not_ happens.

Moreover, my ingestion task is configured to move my realtime data to historical node every day. When this happens, the dimensions are then available for that data, just as when I was loading the same data via batch.

In short, if I wait one day, I can query my once-upon-a-time realtime data properly.

If I would have to bet, I would assume that the API to get the dimensions from druid for the realtime data is different than what we do for historical data.

I am happy to assist in anyway I can.

The other details
------

**Browsers**: Using Safari and Chrome interchangeably.
**OS**: Druid + metabase on Docker (alpine), Client on MacOSX
**Database**: Druid
**Metabase DB**: H2

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with Druid realtime ingestion through the Kafka indexer or Tranquility, then compare dimension visibility before and after data moves to a historical node. Done means dimensions from realtime data appear in table views, filters, and group-by controls without waiting for historical handoff.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.