Improving server segment download code
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
1. Currently if we use the default code path for server segment download we won't retry download on untar failure. In situations as downloading partial tar file, this would not be helpful. There are two solutions:
- improve the default code path in `BaseTableDataManager`
- use the new stream download untar code path `pinot.server.instance.segment.stream.download.untar`
2. The new stream download untar code path currently doesn't support stream decryption, as should be theoretically possible with block cipher. This potential improvement to stream segment download can be explored.
User doc for stream download untar: https://docs.pinot.apache.org/operators/tutorials/performance-optimization-configurations
PR: https://github.com/apache/pinot/pull/8753
Contributor guide
Research direction
Start with BaseTableDataManager and the stream download untar path controlled by pinot.server.instance.segment.stream.download.untar. Read the linked stream download untar user documentation and PR #8753 to understand the existing approaches. Done requires a decided approach for retrying untar failures and determining whether stream decryption should be supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100