apache / apache/pinot

Losing a controller leads to a minimum 5 minute window where segments cannot be committed

Open
#11,488 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

We caught this as we're trying to use the new ingestion lag metric. There is a [5 minute delay](https://github.com/apache/pinot/blob/2f48060d37f94a4570a0b15b13a58391e177ee8a/pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java#L109C41-L109C41) before the lead controller resource rebalances partitions. When we unexpectedly lose a controller, maybe this helps. But Pinot has a path for gracefully shutting down a controller, and this shouldn't impact ingestion lag.

Potential solutions here:
- interestingly enough, helix has added [on demand rebalance](https://github.com/apache/helix/pull/2601) support as of last week. I know we recently upgraded to helix 1.x, so maybe a further upgrade is possible
- we can make the rebalance delay configurable. I think for us, we're just going to test with 0, or no delay. It doesn't seem like rebalancing is particularly expensive, and server retry segment commits quite frequently. This raises a secondary issue that the `leadControllerResource` cannot be update once created, though that's a simple enough fix.

Technically, these aren't mutually exclusive. Ideally we would do both.

Happy to hear other thoughts here or maybe even a background on if 5 minutes was choses for a reason.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.