hiero-ledger / hiero-ledger/hiero-consensus-node

Diagnose and fix stuck gauge for `app_tokenAssociationsPercentUsed`

Open
#13,784 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

On May 24th there was an update to increase the max number of token associations in mainnet. Devops discovered on 7 June that node 7 was reporting an incorrect value, represented in Grafana as `app_tokenAssociationsPercentUsed`:

![Screenshot 2024-06-07 at 10 56 42](https://github.com/hashgraph/hedera-services/assets/112430600/31fd40db-b29d-4d2a-91cc-972daee795bc)

The remainder of the network was in agreement that the percentage dropped, which was expected, but node 7 remained the same. We can also see in the next graph that the max stopped increasing after a brief period of no reporting, which may be related.
![image](https://github.com/hashgraph/hedera-services/assets/112430600/ea5f7ca3-6710-485d-a507-af26ca1eda5d)

After looking at the code that reports the metrics, it isn't immediately obvious what the problem is. The defaults in code for the `tokens.maxAggregateRels` property are appropriately set to 50 million [here](https://github.com/hashgraph/hedera-services/blob/develop/hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/TokensConfig.java#L26-L26); and the code for reporting the size in this metric also seems correct ([here](https://github.com/hashgraph/hedera-services/blob/develop/platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/state/spi/WritableKVStateBase.java#L235-L235)). Furthermore, we pulled the properties files from node 7 and a control node (node 10), but there was no override for `tokens.maxAggregateRels` in either file.

Ultimately, restarting the node seemed to fix the issue. After the restart, node 7's metrics agreed with the rest of the network. We're not sure why. There are a few theories, but nothing concrete:

* The reported size of token relations in state has some sort of bug
* Node07 has had a historical issue where docker containers do not properly start, stop, or restart
* The gauge code, or stats code, has a bug somewhere

We'll need to keep watching this to see if it happens again. If so, we need to document the circumstances. We can also speak with the platform team to see if they have any idea what the issue is.

Contributor guide

Open the contributing guide

Research direction

Start with TokensConfig.java and WritableKVStateBase.java, then inspect how app_tokenAssociationsPercentUsed is reported and compare the node 7 and node 10 properties and restart observations. Done means identifying and fixing a reproducible cause of the stuck gauge, or documenting the circumstances and evidence if the issue cannot be reproduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, grafana, java
Domain
infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.