apache / apache/druid

Add IS_MULTIVALUE and COMPLEX_TYPE columns to INFORMATION_SCHEMA.COLUMNS

Open
#9,707 6 comments 0 reactions 0 assignees View on GitHub
Area - SQL Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

Currently, the `INFORMATION_SCHEMA.COLUMNS` table in Druid does not have full information about the columns in a data source, specifically whether or not a column can contain multi-valued entries and detailed information about which complex serde a metric uses. This means that users attempting to interact with Druid metadata must either fall back to (potentially expensive) SegmentMetadataQueries to determine this information or implement complicated handling logic.

### Motivation

It is currently difficult to obtain complete metadata about a Druid datasource. The only way to determine whether or not a column can contain multiple values or what complex serde was used to encode a metric is to use SegmentMetadataQueries. These queries can be expensive (if aggregating over a large number of segments) and can miss information if not all segments are queried (e.g. columns that don't appear in a given segment won't be returned by a SegmentMetadataQuery, and columns that don't have multiple values _in a queried segment_ will return `..."hasMultipleValues" : false,...`). This can be worked around to some degree via merging results, but this reintroduces the problem that SegmentMetadataQueries can be expensive when run over many segments. If the `INFORMATION_SCHEMA.COLUMNS` table were to be extended to include information about whether a column could contain multiple values and what serde was used for a complex metric, tools that interact with Druid metadata such as the [Calcite Druid adapter](https://calcite.apache.org/docs/druid_adapter.html), proposed Spark and Hive readers, and other third party integrations could issue simple SQL-based queries to determine data source metadata instead of needing to rely on SegmentMetadataQueries. This would also align with the Druid recommendation to use the `INFORMATION_SCHEMA` tables for metadata if you're using SQL.

Contributor guide

Open the contributing guide

Research direction

Begin at the INFORMATION_SCHEMA.COLUMNS entry point and the metadata path used by SegmentMetadataQueries; trace how multi-valued columns and complex serde details are currently represented. Define the expected values for IS_MULTIVALUE and COMPLEX_TYPE, expose them in the SQL schema, and verify them against datasources with multi-valued and complex metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.