Deployment script failure recovery

Occasionally when we run the hardhat script for deploying new contracts on Mainnet, we run into errors.
Root cause varies but can in my experience be attributed to

• A bug in the deployment script (most likely). For example a missing await that does not get caught on local/fork testing due to different in block mining time. Or because the deploy script when run on Mainnet goes thru a slightly different code path vs local/fork.
• Network failure on the host running the deploy (oops my wifi went down while the script was waiting for a tx to get mined!)

In case of such a failure, the operator typically has to manually figure at what step the script failed (using console logs or by checking blockchain txs), comment out the portion of the script already executed, modify the script (for ex. to read addresses of contracts already deployed in the prior failed run) and re-run the script. This can get a bit painful.

It would be nice if we could enhance our current deploy script framework to have some sort of intermediate "checkpoints". And allow to restart where the script failed.

Note: this may complicate things quite a bit in terms of the deploy logic. It may not be worth the extra complexity and engineering time. The alternative is to not worry about partial recovery and just pay the extra gas costs and re-run the entire script... :)

Start by reading the current Hardhat deployment script framework and running the Mainnet deployment flow to understand how failures and already-mined transactions are handled. Define what checkpoint persistence and restart behavior should look like, including how the operator identifies the failed step and confirms recovery is complete.