ElectrumX becoming unresponsive

Nobody has claimed this yet.

Versions

python=3.7.1, 3.9.1
electrumx=1.16.0
aiohttp=3.7.0, 4.0

Context

After running ElectrumX for a few hours, when under very high load (2k+ sessions), sometimes it "locks up".
That is, the main thread blocks and stops making progress, it uses 100% CPU,
and it stops accepting new sessions.
The LocalRPC interface stops responding too; all requests there time out.
It never recovers on its own; the server (python process) must be restarted.

Symptoms

ElectrumX Nonresponsive for tcp, ssl, or rpc services
ElectrumX 100% cpu

Stack Trace

pyrasite-trace.txt

main thread traceback

--- Stack for thread 140338657445696 ---
  File "/electrumx.git/./electrumx_server", line 43, in <module>
    main()
  File "/electrumx.git/./electrumx_server", line 35, in main
    asyncio.run(controller.run())
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
    handle._run()
  File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.9/site-packages/aiorpcx/session.py", line 496, in _throttled_request
    await self._send_message(message)
  File "/usr/local/lib/python3.9/site-packages/aiorpcx/session.py", line 153, in _send_message
    await self.transport.write(message)
  File "/usr/local/lib/python3.9/site-packages/aiorpcx/rawsocket.py", line 118, in write
    self._asyncio_transport.write(framed_message)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 386, in write
    self._ssl_protocol._write_appdata(data)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 598, in _write_appdata
    self._process_write_backlog()
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 674, in _process_write_backlog
    ssldata, offset = self._sslpipe.feed_appdata(data, offset)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 261, in feed_appdata
    offset += self._sslobj.write(view[offset:])
  File "/usr/local/lib/python3.9/ssl.py", line 897, in write
    return self._sslobj.write(data)

Known Workarounds

set up nginx and let it terminate SSL

If one places nginx in front of ElectrumX, and lets nginx handle SSL termination, the problem should not occur.

In such a setup, nginx is reporting these errors:

2021/01/11 02:28:30 [crit] 21#21: *437620 SSL_shutdown() failed (SSL: error:14094123:SSL routines:ssl3_read_bytes:application data after close notify) while proxying connection, client: REDACTED, server: 0.0.0.0:50002, upstream: "127.0.0.1:50213", bytes from/to client:81/205, bytes from/to upstream:205/0

Patch CPython.
- see https://github.com/spesmilo/electrumx/issues/92#issuecomment-832095629
Use uvloop
- the problematic code is in the SSL wrappers in the stdlib asyncio code, which is not used when using uvloop
- set EVENT_LOOP_POLICY=uvloop; this needs uvloop as an additional dependency
  https://electrumx-spesmilo.readthedocs.io/en/latest/environment.html#envvar-EVENT_LOOP_POLICY

Start with the electrumx_server entry point and the attached main-thread traceback, then inspect the asyncio SSL write path implicated in sslproto.py and ssl.py. Reproduce the lockup under high session load and compare behavior with nginx SSL termination or uvloop. Done means TCP, SSL, and LocalRPC services continue accepting and responding without 100% CPU or a restart.

Written by the indexing model from the issue text.

Description

Versions

Context

Symptoms

Stack Trace

Known Workarounds

Contributor guide

First steps

Research direction

Assessment

ElectrumX becoming unresponsive

Description

Versions

Context

Symptoms

Stack Trace

Known Workarounds

Contributor guide

First steps

Research direction

Assessment

Get new issues in your inbox