Abnormal termination with hashXs[hashX].remove(tx_hash) KeyError

I guess there is a bug somewhere that is triggered by particular types of transactions. It's the third time electrumx process terminates abnormally, after running smooth for so much time.

Relevant log:

Jul 01 14:35:34: INFO:Controller:shutting down
Jul 01 14:35:34: INFO:Controller:shutdown complete
Jul 01 14:35:34: ERROR:electrumx:ElectrumX server terminated abnormally
Jul 01 14:35:34: Traceback (most recent call last):
Jul 01 14:35:34:   File "/usr/local/bin/electrumx_server", line 35, in main
Jul 01 14:35:34:     asyncio.run(controller.run())
Jul 01 14:35:34:   File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
Jul 01 14:35:34:     return loop.run_until_complete(main)
Jul 01 14:35:34:   File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
Jul 01 14:35:34:     return future.result()
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/electrumx/lib/server_base.py", line 129, in run
Jul 01 14:35:34:     await server_task
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/electrumx/lib/server_base.py", line 102, in serve
Jul 01 14:35:34:     await self.serve(shutdown_event)
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/electrumx/server/controller.py", line 134, in serve
Jul 01 14:35:34:     await group.spawn(wait_for_catchup())
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/aiorpcx/curio.py", line 297, in __aexit__
Jul 01 14:35:34:     await self.join()
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/electrumx/lib/util.py", line 377, in join
Jul 01 14:35:34:     raise exc
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/electrumx/server/mempool.py", line 366, in keep_synchronized
Jul 01 14:35:34:     await group.spawn(self._logging(synchronized_event))
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/aiorpcx/curio.py", line 297, in __aexit__
Jul 01 14:35:34:     await self.join()
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/electrumx/lib/util.py", line 377, in join
Jul 01 14:35:34:     raise exc
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/electrumx/server/mempool.py", line 255, in _refresh_hashes
Jul 01 14:35:34:     await self._process_mempool(hashes, touched, height)
Jul 01 14:35:34:   File "/home/ss/.local/lib/python3.9/site-packages/electrumx/server/mempool.py", line 281, in _process_mempool
Jul 01 14:35:34:     hashXs[hashX].remove(tx_hash)
Jul 01 14:35:34: KeyError: b'\x88o\x8a1\xca\xf5ap\xfb[\xb1\xe5\xe4%\xcdA~\xe3\x03\xc1\xbf!\xf7\xd7\xe7\x94a7\x0c\xde\xaaj'
Jul 01 14:35:36: electrumx.service: Succeeded.
Jul 01 14:35:36: electrumx.service: Consumed 2d 19h 29min 58.044s CPU time.

What could be a fix for this?

Start in electrumx/server/mempool.py, especially _refresh_hashes and _process_mempool, using the supplied traceback to trace how hashXs and tx_hash are handled. Identify the transaction sequence that causes the KeyError and verify a fix by confirming that mempool synchronization no longer terminates the server abnormally.