Helix Incorrectly Reports Pending Messages (potentially)
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
We use the following MBean to track Helix Pending messages in Pinot Servers:
```
CLMParticipantReport:MonitorType=ParticipantMessageMonitor,ParticipantName=some_instance_id:
```
We have often seen that this MBean reports that there are Helix pending messages, but there's no thread processing any Helix messages. There are also no messages in `INSTANCES//MESSAGES` ZNode.
In this particular instance, I see the following things happened around the time we saw this issue:
* There was a big GC pause
* Zk client lost connection
* There's a Helix log message: `Tasks that never commenced execution after 200`. This is followed by a list of 238 FutureTask, and the MBean is set to exactly 238 Pending Messages. There are also log lines which indicate that the `HelixTaskExecutor` had shutdown.
* There were failures in deleting Helix messages in `INSTANCES//MESSAGES`. There were 9 such messages.
Some other signals can be seen in the attached Grafana screenshot.
Contributor guide
Assessment
This issue has not been assessed yet.