hiero-ledger / hiero-ledger/hiero-consensus-node
Stale events silently drop user transactions
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
### Description
With additional logging to `DefaultStaleEventDetector.addConsensusRound()`to report stale events, I observe:
```
2024-09-05 18:37:17.135 33328 WARN INVALID_EVENT_ERROR DefaultStaleEventDetector: addConsensusRound gen: 110945 created: 2024-09-05T18:37:08.469161362Z #txn: 109 threshold: 110953
```
This means that the event went stale 9 seconds after it was created. There were 109 transactions in the event, mostly user transactions. All those transactions were silently dropped (only system transactions are resubmitted). However, transaction records remained cached and `TransactionReceiptQueries` would return `OK` until transaction expiration - for another 171 seconds. Finally, client gets `RECEIPT_NOT_FOUND`. It has to check the status of the transaction with the mirror node only to find out that it has not been executed, so the transaction needs to be resubmitted by the client. All that creates poor user experience. It also affects performance testing as pending transactions decrease throughput.
### Steps to reproduce
Stale events and their effect were observed in a performance network (`engnet1`) when running the `NftTransferLoadTest` benchmark at ~10K TPS.
### Additional context
_No response_
### Hedera network
other
### Version
v0.54.0-SNAPSHOT
### Operating system
Linux
Contributor guide
Research direction
Start with DefaultStaleEventDetector.addConsensusRound() and trace how stale events containing user transactions are handled, including the cached records observed through TransactionReceiptQueries. Use the NftTransferLoadTest performance scenario to reproduce the behavior; done should address the silent drop and misleading OK receipt state described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100