[Proposal] segmentMetadata query returns full list of dimensions
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Motivation
Currently segmentMetadata query doesn't return dimensions which have no values (cardinality=0) for the time range queried. Some tools (i.e. Imply Pivot) use this type of query to fetch metadata and then validate all user requests against it. When these tools got in such situation with 'non-existing' dimension they treat the request as invalid. Although Druid Broker can process JSON request with such dimension correctly and return an expected result. Even more that these dimensions are displayed in timeline in coordinator views. So there is even inconsistency between it and segmentMetadata query.
Situation got worse after introducing SQL to Druid. For now Druid fails when these dimensions are requested in SQL expression. Example of the error response:
{
"error": "Unknown exception",
"errorMessage": "org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 18: Column 'xxx' not found in any table",
"errorClass": "org.apache.calcite.tools.ValidationException",
"host": null
}
It looks like Apache Calcite tries to follow the described workflow and match request to metadata and receives the same trouble.
### Proposed changes
Change segmentMetadata query to output all dimensions even they do not have data in the requested time range.
### Rationale
It is natural for data storages to have such dimensions with occasional data. The metadata of an data source should be the same for any time range even there are such dimensions in it. Behavior of Druid should be consistent across all its tools and do not change on any cases. Druid core should be consistent and SQL queries should not fail an error for requests which JSON equivalent works as expected.
### Operational impact
No operational impact.
Contributor guide
Research direction
Start by tracing the segmentMetadata query entry point and the metadata handling that determines which dimensions are returned for a time range. Compare its behavior with the JSON query path and the SQL validation behavior described in the issue. Done means segmentMetadata returns dimensions with zero cardinality and SQL requests for those dimensions no longer fail when the equivalent JSON request works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend-api-design, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100