Transaction stuck waiting for finalized in test environment.

We encountered a strange behavior while performing some tests using tokio where sometimes (randomly) the transactions sent to a local test node are stuck endlessly waiting to finalized, even thought the local node keeps on generating blocks.

In our tests, a node is instantiated with manual seal on the same test function where we later perform some transactions and expect them to be finalized by awaiting on wait_for_finalized_success on the received TxProgress.

The problem we are encountering is that after we upgraded the node from 0.9.42 to 1.1.0 or greater and subxt was bumped from 0.26.0 to 0.33.0, we noticed that sometimes the transaction will get stuck waiting for a finalized event forever, even when the local testing chain continues to produce blocks and has indeed finalized the one in which the transaction was included.

Adding extra logs shows that the last status event received is Validated, yet InBestBlock or InFinalizedBlock never arrives.

Furthermore, this only happens in a random manner, and does not happen when interacting with a live chain. We have not been able to find a pattern about which transactions get stuck, yet it seems that it can happen for any transaction sent with this set-up.

Edit: This issue only happens on Linux but works as expected on macOS. Perhaps this hints at something.

Start with the manual-seal setup in testchain/node/src/service.rs around line 694 and the status handling referenced in subxt/src/tx/tx_progress.rs around line 136. Reproduce the transaction wait on Linux using the reported node and subxt version changes, then compare the observed status sequence with macOS. Done means identifying and fixing the condition that prevents InBestBlock or InFinalizedBlock from arriving, with a regression test for the local test-node setup.