apache / apache/druid

make segmentID available in segment metadata and a rowID virtual column

Open
#8,627 0 comments 0 reactions 0 assignees View on GitHub
Area - Querying Design Review Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Motivation

We have use cases that would be equivalent to following standard SQL query
`select * from T where order by x,y,z`

- "select" or "scan" query can't do it as it can't sort on arbitrary columns but just on __time.
- "topN" query can't do it as it supports sort on single column

so, we decided to use the "groupBy" query which supports all the requirements except that it "groups" the rows. there is no natural "row-id" in our data, so we created a `VirtualColumn` that returns `concat(uuid,row-offset-in-segment)` and use that as a dimension in "groupBy" query to prevent "grouping" rows.
however, `uuid` is generated at query time so could be different across multiple runs of same query and produces non-deterministic results when sorting.

Idea is to make available "segmentId" to the `VirtualColumn` interface so that "segmentId" could be used instead of a generated uuid , this will produce deterministic `row-id` for each row in the segment.

However, there might be one slight wrinkle. In Realtime ingestion case, there might not be a natural `segmentId` (when data is in in-memory index or intermediate-persisted segments). But we could try to have as stable as possible an id for that case.

### Proposed changes

Add a new field to `Metadata` class like
```
// Note: it may not be a real segmentId for segments used during realtime indexing
// This is not persisted in segment metadata on disk
@JsonIgnore
private String segmentID;
```

`IndexLoader.load(..)` and various other places would be updated to set `segmentID` in metadata object.

`Metadata` is already accessible to `VirtualColumn` interface, so no changes there. A new `VirtualColumn` implementation would be added to return unique rowIds.

### Rationale

One alternative is to generate a rowID column in segment at time of indexing, but that would be a high cardinality column consuming too much space unnecessarily.

Or, change one of the existing query types either `TopN` to support multiple columns or `GroupBy` not doing the grouping based on some flag. However, proposed option is way simpler and having a virtual `rowID` column might be useful in some other cases as well.

### Operational impact

None

### Test plan (optional)

Unit Tests would be written.

Contributor guide

Open the contributing guide

Research direction

Start with the Metadata class and IndexLoader.load(..), then trace the other places that populate metadata. Review the VirtualColumn interface and design the proposed implementation for deterministic row IDs, including behavior for realtime indexing. Add unit tests covering the segment ID and row ID behavior; done means metadata exposes the ID and the virtual column can use it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.