apache / apache/pinot

[PEP] Add per-index-type storage size breakdown

Open
#19,241 0 comments 0 reactions 0 assignees View on GitHub
PEP-Request
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

## What

Extend `GET /tables/{table}/size` and `GET /tables/{table}/metadata` to expose how much disk space each index type (forward index, inverted index, bloom filter, range index, star-tree, text, vector, dictionary) consumes per replica.

## Problem

[#18185](https://github.com/apache/pinot/pull/18185) added tier-level storage breakdown and per-column compression stats. The total segment size per tier is still a black box — operators cannot tell which index types contribute most to storage cost, making informed index configuration decisions impossible.

## Prior art

`GET /tables/{tableName}/segments/{segmentName}/metadata` already returns per-column, per-index-type sizes (`ColumnMetadataImpl.getIndexSizeMap()`, from [#15591](https://github.com/apache/pinot/pull/15591)) — but only for V3 segments (reads `v3/index_map`) and packed indexes (excludes directory-backed Lucene text/HNSW vector indexes). This PEP extends the concept to V1/V2, directories, and table-level aggregation.

It's also broken for remoteTier segments: `SegmentMetadataImpl.init()` silently no-ops when `v3/index_map` isn't present locally at load time, and tier backends aren't guaranteed to reproduce it. The read path doesn't branch on tier — it's a hidden dependency on local directory completeness. The reload-time refresh below inherited the same dependency and needed a fix.

## Solution

Collect per-index sizes at segment write time and persist to `metadata.properties` — the single source for all segment types (V1/V2/V3, server/minion). The controller fans out via the same bounded path from [#18185](https://github.com/apache/pinot/pull/18185), aggregates by index type, and exposes a new `indexSizeBreakdown` field on existing API responses.

## Collection — write time

No changes to `seal()` ordering. Sizes are collected inside `flushColIndexes()` just before `writeMetadata()`, when all index files exist on disk in V1 format:

```
flushColIndexes()
├─ seal all index creators
├─ collectIndexSizes() ← stat V1 files; store in memory
└─ writeMetadata() ← single write including indexSizeInBytes keys
```

- **Regular index files**: stat using `IndexType.getFileExtensions(colMeta)`. For V3 add `MAGIC_MARKER_SIZE_BYTES` (8) per file — V3 prefixes every packed entry in `columns.psf` with an 8-byte magic marker. Not applied to directories.
- **External Lucene/vector dirs** (`storeInSegmentFile=false`): `TextIndexUtils.hasTextIndex()` / `VectorIndexUtils.hasVectorIndex()` + `FileUtils.sizeOfDirectory()`.

**Persisted to `metadata.properties`:**
```properties
column.message.indexSizeInBytes.inverted_index = 12345678
column.message.indexSizeInBytes.text_index = 34567890
column.user_id.indexSizeInBytes.forward_index = 8901234
```

## Collection — reload time

`SegmentPreProcessor.process()` updates `indexSizeInBytes.*` only for indexes an `IndexHandler` actually adds or removes this reload — it never clears and re-derives every key from scratch. An earlier version did, and that was the root cause of a silent data-loss bug on tiered segments (see Prior art): re-deriving requires the local directory to be byte-for-byte complete (`v3/index_map` for packed entries, individual files otherwise), which a remoteTier segment's tier-materialized copy doesn't guarantee.

- **Added**: size comes from the writer's in-memory record at write time (`SingleFileIndexDirectory`'s `IndexEntry._size`, marker-inclusive for non-v1 targets) or raw file length for v1. Reading it back from the freshly-written `index_map` right after this reload's own packing step is equivalent — it's authored locally, this reload.
- **Removed**: the key is cleared.
- **Untouched** (including a pure tier-migration move with no index changes): left as-is in `metadata.properties` — already correct from an earlier write, nothing to re-derive.

Rule: scope by "did this reload touch it," not by "is `index_map` non-empty." A freshly-written `index_map` only proves the entries just written are correct — it says nothing about untouched entries, which depend on the *old* manifest having loaded correctly at open time, the exact dependency this fix avoids. Format-agnostic (V1/V2/V3 identical), no tier detection. `metadata.properties` is the single read source for untouched entries. Doesn't affect CRC (`.fwd`/`.dict` only).

## API response

```json
"indexSizeBreakdown": {
"forward_index": { "sizePerReplicaInBytes": 32000000000 },
"inverted_index": { "sizePerReplicaInBytes": 18000000000 },
"bloom_filter": { "sizePerReplicaInBytes": 2000000000 },
"star_tree": { "sizePerReplicaInBytes": 4000000000 },
"dictionary": { "sizePerReplicaInBytes": 1500000000 },
"text_index": { "sizePerReplicaInBytes": 8000000000 },
"vector_index": { "sizePerReplicaInBytes": 12000000000 }
}
```

`indexSizeBreakdown` is a table-level aggregate — it does not respect the `columns=` filter, consistent with `columnCompressionStats` from [#18185](https://github.com/apache/pinot/pull/18185). The existing `columnIndexSizeMap` already provides per-column filtered sizes for loaded segments; `indexSizeBreakdown` adds V1/V2, directory-backed indexes, and table-level aggregation.

## Flag

`tableIndexConfig.indexSizeStatsEnabled` (default `false`) gates both collection and API inclusion. `indexSizeBreakdown` appears in responses only when `?includeIndexSizeStats=true` is passed — same pattern as `?includeColumnStats=true` in [#18185](https://github.com/apache/pinot/pull/18185).

## Modules touched

| Module | Change |
|---|---|
| `pinot-segment-spi` | New `metadata.properties` key constants; new accessor in `ColumnMetadata` |
| `pinot-segment-local` | `collectIndexSizes()` in `flushColIndexes()`; reload update in `SegmentPreProcessor` |
| `pinot-common` | New `IndexSizeBreakdownInfo` DTO |
| `pinot-server` | Read from `ColumnMetadataImpl`, include in response |
| `pinot-controller` | Aggregate by index type, expose on existing APIs |

## Notes

- `forwardIndexAndDictionaryStorageSizePerReplicaInBytes` from [#18185](https://github.com/apache/pinot/pull/18185) combines forward index and dictionary for compression ratio. `indexSizeBreakdown` breaks them out separately for a complete per-index-type cost picture.
- Pinot Console UI changes and backfilling existing segments are out of scope.

## Implementation
[#19255](https://github.com/apache/pinot/pull/19255)

Contributor guide

Open the contributing guide

Research direction

Start with the listed pinot-segment-spi, pinot-segment-local, pinot-common, pinot-server, and pinot-controller modules. Read collectIndexSizes() in flushColIndexes(), the SegmentPreProcessor reload path, and the existing table size and metadata API aggregation from #18185. Done means gated index-size collection and indexSizeBreakdown responses work across the stated segment formats and index types; the implementation link should be checked before starting.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.