hiero-ledger / hiero-ledger/hiero-consensus-node

Quiescence doesn't start for inactive network

Open
#27,264 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

## Summary

This behavior was observed in `v0.75.1` with `blockStream.streamMode=RECORDS`, which is the default setting for Solo. Quiescence appears to be working as intended for `blockStream.streamMode=BLOCKS`

After enabling `quiescence.enabled=true` on a Solo-deployed network and driving load through it with a transaction pinger, the network correctly breaks quiescence under load but never re-enters QUIESCE once the load stops. `hgcaa.log` shows that the node goes back and forth between `QUIESCE` and `DONT_QUIESCE`, but always settles in `DONT_QUIESCE`, even when there are no submitted transaction:

```
...
2026-09-14 08:47:43.790 INFO 92 QuiescedHeartbeat - Started quiesced heartbeat at interval PT1S
2026-09-14 08:47:43.790 INFO 128 QuiescedHeartbeat - Stopping quiescence heartbeat (DONT_QUIESCE)
2026-09-14 08:47:43.850 INFO 692 BlockRecordManagerImpl - Updating quiescence command from QUIESCE to DONT_QUIESCE
2026-09-14 08:47:44.990 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to QUIESCE
2026-09-14 08:47:44.990 INFO 92 QuiescedHeartbeat - Started quiesced heartbeat at interval PT1S
2026-09-14 08:47:44.990 INFO 128 QuiescedHeartbeat - Stopping quiescence heartbeat (DONT_QUIESCE)
2026-09-14 08:47:45.050 INFO 692 BlockRecordManagerImpl - Updating quiescence command from QUIESCE to DONT_QUIESCE
2026-09-14 08:47:46.250 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to BREAK_QUIESCENCE
2026-09-14 08:47:46.309 INFO 692 BlockRecordManagerImpl - Updating quiescence command from BREAK_QUIESCENCE to DONT_QUIESCE
2026-09-14 08:47:47.450 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to BREAK_QUIESCENCE
2026-09-14 08:47:47.510 INFO 692 BlockRecordManagerImpl - Updating quiescence command from BREAK_QUIESCENCE to DONT_QUIESCE
2026-09-14 08:47:48.649 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to BREAK_QUIESCENCE
2026-09-14 08:47:48.709 INFO 692 BlockRecordManagerImpl - Updating quiescence command from BREAK_QUIESCENCE to DONT_QUIESCE
2026-09-14 08:47:49.851 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to BREAK_QUIESCENCE
2026-09-14 08:47:49.910 INFO 692 BlockRecordManagerImpl - Updating quiescence command from BREAK_QUIESCENCE to DONT_QUIESCE
2026-09-14 08:47:51.051 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to QUIESCE
2026-09-14 08:47:51.051 INFO 92 QuiescedHeartbeat - Started quiesced heartbeat at interval PT1S
2026-09-14 08:47:51.051 INFO 128 QuiescedHeartbeat - Stopping quiescence heartbeat (DONT_QUIESCE)
2026-09-14 08:47:51.110 INFO 692 BlockRecordManagerImpl - Updating quiescence command from QUIESCE to DONT_QUIESCE
2026-09-14 08:47:52.309 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to BREAK_QUIESCENCE
2026-09-14 08:47:52.369 INFO 692 BlockRecordManagerImpl - Updating quiescence command from BREAK_QUIESCENCE to DONT_QUIESCE
2026-09-14 08:47:53.510 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to BREAK_QUIESCENCE
2026-09-14 08:47:53.569 INFO 692 BlockRecordManagerImpl - Updating quiescence command from BREAK_QUIESCENCE to DONT_QUIESCE
2026-09-14 08:47:54.771 INFO 692 BlockRecordManagerImpl - Updating quiescence command from DONT_QUIESCE to QUIESCE
2026-09-14 08:47:54.771 INFO 92 QuiescedHeartbeat - Started quiesced heartbeat at interval PT1S
2026-09-14 08:47:54.771 INFO 128 QuiescedHeartbeat - Stopping quiescence heartbeat (DONT_QUIESCE)
2026-09-14 08:47:54.831 INFO 692 BlockRecordManagerImpl - Updating quiescence command from QUIESCE to DONT_QUIESCE
```

## Steps to reproduce

```
cat > ./quiescence-application.properties <<'EOF'
quiescence.enabled=true
EOF
cat > ./falcon-quiescence-values.yaml <<'EOF'
network:
--application-properties: quiescence-application.properties
EOF
solo one-shot falcon deploy --deployment quiescence-repro --namespace quiescence-repro --values-file ./falcon-quiescence-values.yaml
```

Send transactions with: `solo ledger account create --deployment quiescence-repro`

When the deployment is created observe the quiescence heartbeat logs:
```
kubectl exec -n quiescence-repro network-node1-0 -c root-container -- tail -f /opt/hgcapp/services-hedera/HapiApp2.0/output/hgcaa.log
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with quiescence-application.properties and falcon-quiescence-values.yaml, then inspect hgcaa.log from the network-node container. Start with the QuiescedHeartbeat and BlockRecordManagerImpl log entries, comparing RECORDS with BLOCKS mode. Done means an inactive RECORDS-mode network re-enters QUIESCE after transactions stop instead of settling in DONT_QUIESCE.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.