hiero-ledger / hiero-ledger/hiero-consensus-node

Stale events silently drop user transactions

Open
#15,366 3 comments 0 reactions 0 assignees View on GitHub
New Feature Platform
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

### Description

With additional logging to `DefaultStaleEventDetector.addConsensusRound()`to report stale events, I observe:
```
2024-09-05 18:37:17.135 33328 WARN INVALID_EVENT_ERROR DefaultStaleEventDetector: addConsensusRound gen: 110945 created: 2024-09-05T18:37:08.469161362Z #txn: 109 threshold: 110953
```
This means that the event went stale 9 seconds after it was created. There were 109 transactions in the event, mostly user transactions. All those transactions were silently dropped (only system transactions are resubmitted). However, transaction records remained cached and `TransactionReceiptQueries` would return `OK` until transaction expiration - for another 171 seconds. Finally, client gets `RECEIPT_NOT_FOUND`. It has to check the status of the transaction with the mirror node only to find out that it has not been executed, so the transaction needs to be resubmitted by the client. All that creates poor user experience. It also affects performance testing as pending transactions decrease throughput.

### Steps to reproduce

Stale events and their effect were observed in a performance network (`engnet1`) when running the `NftTransferLoadTest` benchmark at ~10K TPS.

### Additional context

_No response_

### Hedera network

other

### Version

v0.54.0-SNAPSHOT

### Operating system

Linux

Contributor guide

Open the contributing guide

Research direction

Start with DefaultStaleEventDetector.addConsensusRound() and trace how stale events containing user transactions are handled, including the cached records observed through TransactionReceiptQueries. Use the NftTransferLoadTest performance scenario to reproduce the behavior; done should address the silent drop and misleading OK receipt state described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.