apache / apache/pinot

Follower server redownloads segments upon server restart due to CRC change

Open
#11,004 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

When restarting servers in our clusters running on 0.12+, we observe numerous Helix pending messages due to `Failed to Load LLC Segment`. This is caused by CRC mismatch between ZK and the starting server.

Initial investigation showed that the 'leader' server commits and updates ZK, while the 'follower' server catches up and builds the segment locally with a different CRC. When a server is restarted all the segments that it 'followed' fail to load and are redownloaded from our deep store. We've confirmed that startOffset/endOffset match and the difference between two segments lies in the `columns.psf` file.

Logs for a segment:
```
2023-06-29T13:45:25-07:00 [host] Adding segment: to table:
2023-06-29T13:45:25-07:00 [host] Segment: of table: has crc change from: 3828021013 to: 1125725625
2023-06-29T13:45:25-07:00 [host] Failed to load LLC segment: , downloading a new copy
```

This behavior isn't seen on our clusters running on a 0.11 base. Is it possible some non-deterministic was introduced in the segment build process?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the restart behavior on the reported 0.12+ and 0.11 clusters, using the “Failed to Load LLC Segment” logs and comparing the CRCs of corresponding segments. Inspect the differing columns.psf file while confirming that startOffset and endOffset match; done means follower-built segments have matching CRCs and are not unnecessarily redownloaded after restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.