Transaction broadcast succeeded but was not included on-chain

Software Versions

OS : Linux
JVM : Oracle Corporation 1.8.0_161 amd64
Version : 4.8.0

Expected behaviour

After a transaction is broadcast successfully, it should be packaged into a block and included on-chain normally.

Actual behaviour

An abnormal behavior was observed in a private network: transactions that were broadcast successfully were not included on-chain.

• The private network consists of 1 SR node and 2 Fullnodes, all connected to each other.
• The two Fullnodes were started at the same time and immediately began synchronizing blocks from the SR (both were in the block-catching-up state).
• During the catch-up process, a transaction was broadcast to one of the Fullnodes:
  • The Fullnode returned a transaction broadcast success response.
  • However, the transaction never included on-chain, and it did not appear in any subsequent blocks.
• Based on network logs alone, the transaction seemed to be received and propagated by the nodes during the catch-up period, but it was not selected by the SR for block production.

Once block syncing completed, the issue disappeared and the system returned to normal behaviour.

Frequency

This issue had not occurred previously. It was discovered during a recent private network deployment and testing.

Start by reproducing the private-network setup with one SR node and two Fullnodes while both Fullnodes are catching up, then review the network logs around transaction broadcast and block production. Done means identifying why the successfully broadcast transaction is not selected for a block during synchronization and confirming normal inclusion after the cause is addressed.