apache / apache/pinot

Stop retrying downloading if a segment if zk metadata is missing

Open
#10,806 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

During the investigation described in https://github.com/apache/pinot/pull/10804, we found the download code in SegmentFetcher would still retry even if the segment metadata is missing, this could make the thread to get held up wastefully. We should add a piece of code to check on the existence of the segment in zk before every time we retry.

Contributor guide

Open the contributing guide

Research direction

Start in the SegmentFetcher download and retry path, using the investigation in pull request 10804 as context. Trace how segment metadata is checked in ZooKeeper before each retry. Done means retries stop when the segment metadata is missing, while normal retries remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.