apache / apache/pinot

Improving server segment download code

Open
#9,152 14 comments 1 reaction 0 assignees View on GitHub
good first issue
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

1. Currently if we use the default code path for server segment download we won't retry download on untar failure. In situations as downloading partial tar file, this would not be helpful. There are two solutions:
- improve the default code path in `BaseTableDataManager`
- use the new stream download untar code path `pinot.server.instance.segment.stream.download.untar`

2. The new stream download untar code path currently doesn't support stream decryption, as should be theoretically possible with block cipher. This potential improvement to stream segment download can be explored.

User doc for stream download untar: https://docs.pinot.apache.org/operators/tutorials/performance-optimization-configurations
PR: https://github.com/apache/pinot/pull/8753

Contributor guide

Open the contributing guide

Research direction

Start with BaseTableDataManager and the stream download untar path controlled by pinot.server.instance.segment.stream.download.untar. Read the linked stream download untar user documentation and PR #8753 to understand the existing approaches. Done requires a decided approach for retrying untar failures and determining whether stream decryption should be supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.