apache / apache/druid

[Proposal] segmentMetadata query returns full list of dimensions

Open
#7,583 5 comments 7 reactions 0 assignees View on GitHub
Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation

Currently segmentMetadata query doesn't return dimensions which have no values (cardinality=0) for the time range queried. Some tools (i.e. Imply Pivot) use this type of query to fetch metadata and then validate all user requests against it. When these tools got in such situation with 'non-existing' dimension they treat the request as invalid. Although Druid Broker can process JSON request with such dimension correctly and return an expected result. Even more that these dimensions are displayed in timeline in coordinator views. So there is even inconsistency between it and segmentMetadata query.
Situation got worse after introducing SQL to Druid. For now Druid fails when these dimensions are requested in SQL expression. Example of the error response:
{
"error": "Unknown exception",
"errorMessage": "org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 18: Column 'xxx' not found in any table",
"errorClass": "org.apache.calcite.tools.ValidationException",
"host": null
}
It looks like Apache Calcite tries to follow the described workflow and match request to metadata and receives the same trouble.

### Proposed changes

Change segmentMetadata query to output all dimensions even they do not have data in the requested time range.

### Rationale

It is natural for data storages to have such dimensions with occasional data. The metadata of an data source should be the same for any time range even there are such dimensions in it. Behavior of Druid should be consistent across all its tools and do not change on any cases. Druid core should be consistent and SQL queries should not fail an error for requests which JSON equivalent works as expected.

### Operational impact

No operational impact.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the segmentMetadata query entry point and the metadata handling that determines which dimensions are returned for a time range. Compare its behavior with the JSON query path and the SQL validation behavior described in the issue. Done means segmentMetadata returns dimensions with zero cardinality and SQL requests for those dimensions no longer fail when the equivalent JSON request works.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend-api-design, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.