apache / apache/pinot

Tagging and Untagging servers can lead to an unexpected state

Open
#19,240 7 comments 0 reactions 0 assignees View on GitHub
troubleshooting
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

**Background:**
I tagged a few servers with a tenant ID (Cell19) but after a while I realized that I had incorrectly tagged some servers as the actual AZ distribution did not match with the expected AZ distribution. So i untagged a few servers but I retagged other servers to have the right AZ distribution.

**Issue:**
This is leading to a state wherein the Helix/Zk thinks that the older servers are still part of the tenant but the server does not have the old tenants tag on it.
The new servers have the tenants tag but theyre not actually ingesting anything.

Image

List of instances on the tenant page

Image

Server that was added later on( this is not doing any work)

Image

Server that was removed from the tenant. As you can see, it does not have the tenant tag (tenant was 'Cell19')

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported sequence: tag servers for tenant Cell19, untag some, and retag others with a different AZ distribution. Inspect the tenant page, Helix/ZK state, server tags, and ingestion status to identify why removed servers remain assigned while newly tagged servers do no work; done means tenant membership and ingestion match the current tags.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.