apache / apache/lucene

Reimplementation of Disk Usage API

Open
#12,395 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

There is an opportunity to improve functionality and performance of existing Disk Usage API, through a re-implementation.

Currently, the best tool we have for this is based on a custom Codec that separates storage by field; to get the statistics we read an existing index and write it out using `AddIndexes` and force-merging, using the custom codec. This is time-consuming and inefficient and tends not to get done.

What we could do is estimate the storage of each field by iterating its structures (i.e., inverted index, doc-values, stored fields, etc.) and tracking the number of read-bytes. Since we will enumerate the index, it wouldn't require us to force-merge all the data through `addIndexes`, and at the same time it doesn't invade the codec apis.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing Disk Usage API, its custom Codec path, and the use of AddIndexes and force-merging. The intended result is a reimplementation that estimates per-field storage by iterating index structures and tracking read-bytes without requiring force-merging or changes to codec APIs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.