Reimplementation of Disk Usage API
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
There is an opportunity to improve functionality and performance of existing Disk Usage API, through a re-implementation.
Currently, the best tool we have for this is based on a custom Codec that separates storage by field; to get the statistics we read an existing index and write it out using `AddIndexes` and force-merging, using the custom codec. This is time-consuming and inefficient and tends not to get done.
What we could do is estimate the storage of each field by iterating its structures (i.e., inverted index, doc-values, stored fields, etc.) and tracking the number of read-bytes. Since we will enumerate the index, it wouldn't require us to force-merge all the data through `addIndexes`, and at the same time it doesn't invade the codec apis.
Contributor guide
Research direction
Start by locating the existing Disk Usage API, its custom Codec path, and the use of AddIndexes and force-merging. The intended result is a reimplementation that estimates per-field storage by iterating index structures and tracking read-bytes without requiring force-merging or changes to codec APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100