hiero-ledger / hiero-ledger/hiero-consensus-node
Diagnose and fix stuck gauge for `app_tokenAssociationsPercentUsed`
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
On May 24th there was an update to increase the max number of token associations in mainnet. Devops discovered on 7 June that node 7 was reporting an incorrect value, represented in Grafana as `app_tokenAssociationsPercentUsed`:

The remainder of the network was in agreement that the percentage dropped, which was expected, but node 7 remained the same. We can also see in the next graph that the max stopped increasing after a brief period of no reporting, which may be related.

After looking at the code that reports the metrics, it isn't immediately obvious what the problem is. The defaults in code for the `tokens.maxAggregateRels` property are appropriately set to 50 million [here](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/TokensConfig.java#L26-L26); and the code for reporting the size in this metric also seems correct ([here](https://github.com/hashgraph/hedera-services/blob/develop/platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/state/spi/WritableKVStateBase.java#L235-L235)). Furthermore, we pulled the properties files from node 7 and a control node (node 10), but there was no override for `tokens.maxAggregateRels` in either file.
Ultimately, restarting the node seemed to fix the issue. After the restart, node 7's metrics agreed with the rest of the network. We're not sure why. There are a few theories, but nothing concrete:
* The reported size of token relations in state has some sort of bug
* Node07 has had a historical issue where docker containers do not properly start, stop, or restart
* The gauge code, or stats code, has a bug somewhere
We'll need to keep watching this to see if it happens again. If so, we need to document the circumstances. We can also speak with the platform team to see if they have any idea what the issue is.
Contributor guide
Research direction
Start with TokensConfig.java and WritableKVStateBase.java, then inspect how app_tokenAssociationsPercentUsed is reported and compare the node 7 and node 10 properties and restart observations. Done means identifying and fixing a reproducible cause of the stuck gauge, or documenting the circumstances and evidence if the issue cannot be reproduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, grafana, java
- Domain
- infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100