lbry-sdk suite: test_es_sync_utility: AssertionError: 217 != 218

It seems this is a vulnerability in the es_writer.start() where it can fail if interleaved with a task adding new block(s) (e.g. create_task(self.generate(N))). Hence filing against lbryio/hub.

Test Code:

        # stop the es writer and advance the chain by 1, adding a new claim. upon resuming the es writer, it should
        # add the new claim
        await es_writer.stop()
        await self.stream_create(f"stream11", bid='0.001', confirm=False)
        generate_block_task = asyncio.create_task(self.generate(1))
        await es_writer.start()
        await generate_block_task
        self.assertEqual(11, len(await self.claim_search(order_by=['height'])))

https://github.com/lbryio/lbry-sdk/blob/8becf1f69f38019c8c1d0ac6fbba80897f94c8ed/tests/integration/blockchain/test_wallet_server_sessions.py#L126

Failure:

======================================================================
[653](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:654)
FAIL: test_es_sync_utility (integration.blockchain.test_wallet_server_sessions.TestESSync)
[654](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:655)
----------------------------------------------------------------------
[655](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:656)
Traceback (most recent call last):
[656](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:657)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/testcase.py", line 145, in run
[657](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:658)
    self.loop.run_until_complete(maybe_coroutine)
[658](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:659)
  File "/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
[659](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:660)
    return future.result()
[660](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:661)
  File "/home/runner/work/lbry-sdk/lbry-sdk/tests/integration/blockchain/test_wallet_server_sessions.py", line 131, in test_es_sync_utility
[661](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:662)
    await es_writer.start()
[662](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:663)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/elastic_sync/service.py", line 389, in start
[663](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:664)
    return await super().start()
[664](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:665)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/service.py", line 81, in start
[665](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:666)
    await start_task
[666](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:667)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/elastic_sync/service.py", line 315, in catch_up
[667](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:668)
    self.advance(height)
[668](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:669)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/elastic_sync/service.py", line 230, in advance
[669](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:670)
    super().advance(height)
[670](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:671)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/service.py", line 159, in advance
[671](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:672)
    assert len(self.db.tx_counts) == height, f"{len(self.db.tx_counts)} != {height}"
[672](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:673)
AssertionError: 217 != 218
[673](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:674)

Examples:
https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:686
https://github.com/lbryio/lbry-sdk/runs/6846528407?check_suite_focus=true#step:11:499
https://github.com/lbryio/lbry-sdk/runs/6846200096?check_suite_focus=true#step:11:469

Reproduce test_es_sync_utility in tests/integration/blockchain/test_wallet_server_sessions.py, focusing on the interleaving of es_writer.stop(), generate(1), and es_writer.start(). Trace the reported path through hub/elastic_sync/service.py catch_up and advance into hub/service.py. Done means the race no longer produces the 217 != 218 assertion and the integration test passes.