apache / apache/pinot

Helix Incorrectly Reports Pending Messages (potentially)

Open
#11,626 3 comments 0 reactions 0 assignees View on GitHub
bug metrics
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

We use the following MBean to track Helix Pending messages in Pinot Servers:

```
CLMParticipantReport:MonitorType=ParticipantMessageMonitor,ParticipantName=some_instance_id:
```

We have often seen that this MBean reports that there are Helix pending messages, but there's no thread processing any Helix messages. There are also no messages in `INSTANCES//MESSAGES` ZNode.

In this particular instance, I see the following things happened around the time we saw this issue:

* There was a big GC pause
* Zk client lost connection
* There's a Helix log message: `Tasks that never commenced execution after 200`. This is followed by a list of 238 FutureTask, and the MBean is set to exactly 238 Pending Messages. There are also log lines which indicate that the `HelixTaskExecutor` had shutdown.
* There were failures in deleting Helix messages in `INSTANCES//MESSAGES`. There were 9 such messages.

Some other signals can be seen in the attached Grafana screenshot.

image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.