Migrate away from submitAndWatch for Transactions and monitor their progress manually?

Currently we use author_submitAndWatchExtrinsic (Legacybackend) or transactionWatch_v1_submitAndWatch (UnstableBackend) to submit transactions and monitor their progress. These calls both give back events detaililng the status of the transaction until it enteres a finalized block or errors out.

An alternative would be to use author_submitExtrinsic (LegacyBackend) or transaction_v1_broadcast (UnstableBackend) to submit an extrinsic. These don't provide any events about the progress of the extrinsic, but it should be possible to recreate these events by subscribing to finalized/all blocks upon submitting transactions, downloading each block body and looking for the transaction in the block. This way, we can see if it enters a current best block, new block or finalized block. To know whether it's errored out, I think we'd also need to do a dryRun runtime API call (https://github.com/paritytech/json-rpc-interface-spec/issues/55) on each finalized block that we don't find the transaction in, allowing us to return an error if not. At least in the case of the new APIs, we could eventually call transaction_v1_stop to give up if need be.

There are pros and cons to both the current (1) and alternative (2) approach:

Pros of (1):

• Fairly network/perf efficient for the client when interacting with an RPC node: no need to download potentially MBs of block bodies searching for the transaction, or making runtime calls to check that it's still valid. We get back events which tell us what we need to know.

Cons of (1):

• If we are disconnected from the RPC node, we will lose these events with no way to recover or resume them on reconnection (I think this isn't an issue if we are using a light client).
• Using the V2 RPCs, there is not a link between when we are told about blocks versus when we receive transaction events about blocks. If we want to return transaction events with block hashes that are guaranteed to be pinned (ie available), we need to manually synchronise these two streams (finalized blocks versus tx events). We can't give back pinned refs for new/best blocks since we don't want to hold up handing back tx events while waiting for hashes which might never be seen if pruned etc.

Pros of (2):

• We can resume providing the user TX events after small disconnections (since we can see recently finalized blocks when re subscribing and continue our search where we left off.
• We can guarantee that all block hashes handed backj to the user in our TX events reference pinned blocks, since we are subscribing to the blocks (and thus pinning them) ourselves in order to emit these events.

Cons of (2):

• Requires downloading block bodies and making runtime API calls in order to reproduce the current TX events that we hand back to users (and even then we may get back less granularity).

One option might be to take a hybrid approach, ie using approach (1) as we do now, but on disconnect pivoting to scanning finalized block bodies and periodic dry running to determine whether the TX made it in. This reduces network traffic in the common case and allows resuming on disconnect, but adds some complexity as a result. Alternately, if we go to the effort of implementing both approaches, we could make it a user choice which one to take, and possibly then default to (2) for maximum robustness at greater network/decode cost.

Start by tracing the current use of author_submitAndWatchExtrinsic and transactionWatch_v1_submitAndWatch, then compare the author_submitExtrinsic and transaction_v1_broadcast APIs described here. The issue does not identify files, tests, or a settled approach; work would be complete only after the monitoring strategy and expected transaction events are agreed and implemented.