apache / apache/druid

rework segment metadata query "size" analysis

Open
#7,124 4 comments 0 reactions 0 assignees View on GitHub
Proposal
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

# Motivation

Currently segment metadata query has a `size` analysis type that the documentation describes as "estimated byte size for the segment columns if they were stored in a flat format" and "estimated total segment byte size in if it was stored in a flat format", but this doesn't really have any practical value as far as I can tell. I think this value is confusing, and should be replaced with a `size` value that represents the actual segment and column sizes for mapped segments, or the estimated size in memory for incremental segments.

This allows size analysis to be useful for finding which columns are the heavy hitters in terms of overall segment size, observe fluctuations in segment size over time, and could also be of aid in capacity planning.

Alternatively, if this doesn't seem useful enough, or more trouble than it's worth, I would instead propose that the value just be removed completely because it's confusing and expensive to run.

# Proposed changes

Making the value meaningful likely doesn't take much effort. For mapped columns, it would just be preserving the byte size of the columns when initially loading segments and making it accessible through the `BaseColumn` or related interface (if it isn't already and I just missed it). For incremental index, it would involve modifying the size functions to report the estimated size in memory of the values where necessary. No surface level changes would be necessary for this approach, but we would need to call out in the release notes that the meanings of values has changed and update documentation accordingly.

# Rationale

We already need to know this size information to properly map columns from the 'smoosh' file, so preserving it and offering it up for segment metadata queries should be rather straightforward.

# Operational impact

Size analysis as is looks rather expensive, if we switch to column "in segment" size, then this operation can become constant for mapped segments resulting in cheaper overall segment metadata queries using this analysis on mapped segments. I expect little change for the case of incremental indexes.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the segment metadata query's existing `size` analysis and inspect the `BaseColumn` or related interface, mapped-column loading, and incremental-index size functions. Determine whether mapped and incremental segments can expose the proposed sizes consistently; done means implementing the chosen semantics and updating the documentation and release notes, or removing the analysis if that approach is rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.