[PEP] Add per-index-type storage size breakdown
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
## What
Extend `GET /tables/{table}/size` and `GET /tables/{table}/metadata` to expose how much disk space each index type (forward index, inverted index, bloom filter, range index, star-tree, text, vector, dictionary) consumes per replica.
## Problem
[#18185](https://github.com/apache/pinot/pull/18185) added tier-level storage breakdown and per-column compression stats. The total segment size per tier is still a black box — operators cannot tell which index types contribute most to storage cost, making informed index configuration decisions impossible.
## Prior art
`GET /tables/{tableName}/segments/{segmentName}/metadata` already returns per-column, per-index-type sizes (`ColumnMetadataImpl.getIndexSizeMap()`, from [#15591](https://github.com/apache/pinot/pull/15591)) — but only for V3 segments (reads `v3/index_map`) and packed indexes (excludes directory-backed Lucene text/HNSW vector indexes). This PEP extends the concept to V1/V2, directories, and table-level aggregation.
It's also broken for remoteTier segments: `SegmentMetadataImpl.init()` silently no-ops when `v3/index_map` isn't present locally at load time, and tier backends aren't guaranteed to reproduce it. The read path doesn't branch on tier — it's a hidden dependency on local directory completeness. The reload-time refresh below inherited the same dependency and needed a fix.
## Solution
Collect per-index sizes at segment write time and persist to `metadata.properties` — the single source for all segment types (V1/V2/V3, server/minion). The controller fans out via the same bounded path from [#18185](https://github.com/apache/pinot/pull/18185), aggregates by index type, and exposes a new `indexSizeBreakdown` field on existing API responses.
## Collection — write time
No changes to `seal()` ordering. Sizes are collected inside `flushColIndexes()` just before `writeMetadata()`, when all index files exist on disk in V1 format:
```
flushColIndexes()
├─ seal all index creators
├─ collectIndexSizes() ← stat V1 files; store in memory
└─ writeMetadata() ← single write including indexSizeInBytes keys
```
- **Regular index files**: stat using `IndexType.getFileExtensions(colMeta)`. For V3 add `MAGIC_MARKER_SIZE_BYTES` (8) per file — V3 prefixes every packed entry in `columns.psf` with an 8-byte magic marker. Not applied to directories.
- **External Lucene/vector dirs** (`storeInSegmentFile=false`): `TextIndexUtils.hasTextIndex()` / `VectorIndexUtils.hasVectorIndex()` + `FileUtils.sizeOfDirectory()`.
**Persisted to `metadata.properties`:**
```properties
column.message.indexSizeInBytes.inverted_index = 12345678
column.message.indexSizeInBytes.text_index = 34567890
column.user_id.indexSizeInBytes.forward_index = 8901234
```
## Collection — reload time
`SegmentPreProcessor.process()` updates `indexSizeInBytes.*` only for indexes an `IndexHandler` actually adds or removes this reload — it never clears and re-derives every key from scratch. An earlier version did, and that was the root cause of a silent data-loss bug on tiered segments (see Prior art): re-deriving requires the local directory to be byte-for-byte complete (`v3/index_map` for packed entries, individual files otherwise), which a remoteTier segment's tier-materialized copy doesn't guarantee.
- **Added**: size comes from the writer's in-memory record at write time (`SingleFileIndexDirectory`'s `IndexEntry._size`, marker-inclusive for non-v1 targets) or raw file length for v1. Reading it back from the freshly-written `index_map` right after this reload's own packing step is equivalent — it's authored locally, this reload.
- **Removed**: the key is cleared.
- **Untouched** (including a pure tier-migration move with no index changes): left as-is in `metadata.properties` — already correct from an earlier write, nothing to re-derive.
Rule: scope by "did this reload touch it," not by "is `index_map` non-empty." A freshly-written `index_map` only proves the entries just written are correct — it says nothing about untouched entries, which depend on the *old* manifest having loaded correctly at open time, the exact dependency this fix avoids. Format-agnostic (V1/V2/V3 identical), no tier detection. `metadata.properties` is the single read source for untouched entries. Doesn't affect CRC (`.fwd`/`.dict` only).
## API response
```json
"indexSizeBreakdown": {
"forward_index": { "sizePerReplicaInBytes": 32000000000 },
"inverted_index": { "sizePerReplicaInBytes": 18000000000 },
"bloom_filter": { "sizePerReplicaInBytes": 2000000000 },
"star_tree": { "sizePerReplicaInBytes": 4000000000 },
"dictionary": { "sizePerReplicaInBytes": 1500000000 },
"text_index": { "sizePerReplicaInBytes": 8000000000 },
"vector_index": { "sizePerReplicaInBytes": 12000000000 }
}
```
`indexSizeBreakdown` is a table-level aggregate — it does not respect the `columns=` filter, consistent with `columnCompressionStats` from [#18185](https://github.com/apache/pinot/pull/18185). The existing `columnIndexSizeMap` already provides per-column filtered sizes for loaded segments; `indexSizeBreakdown` adds V1/V2, directory-backed indexes, and table-level aggregation.
## Flag
`tableIndexConfig.indexSizeStatsEnabled` (default `false`) gates both collection and API inclusion. `indexSizeBreakdown` appears in responses only when `?includeIndexSizeStats=true` is passed — same pattern as `?includeColumnStats=true` in [#18185](https://github.com/apache/pinot/pull/18185).
## Modules touched
| Module | Change |
|---|---|
| `pinot-segment-spi` | New `metadata.properties` key constants; new accessor in `ColumnMetadata` |
| `pinot-segment-local` | `collectIndexSizes()` in `flushColIndexes()`; reload update in `SegmentPreProcessor` |
| `pinot-common` | New `IndexSizeBreakdownInfo` DTO |
| `pinot-server` | Read from `ColumnMetadataImpl`, include in response |
| `pinot-controller` | Aggregate by index type, expose on existing APIs |
## Notes
- `forwardIndexAndDictionaryStorageSizePerReplicaInBytes` from [#18185](https://github.com/apache/pinot/pull/18185) combines forward index and dictionary for compression ratio. `indexSizeBreakdown` breaks them out separately for a complete per-index-type cost picture.
- Pinot Console UI changes and backfilling existing segments are out of scope.
## Implementation
[#19255](https://github.com/apache/pinot/pull/19255)
Contributor guide
Research direction
Start with the listed pinot-segment-spi, pinot-segment-local, pinot-common, pinot-server, and pinot-controller modules. Read collectIndexSizes() in flushColIndexes(), the SegmentPreProcessor reload path, and the existing table size and metadata API aggregation from #18185. Done means gated index-size collection and indexSizeBreakdown responses work across the stated segment formats and index types; the implementation link should be checked before starting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100