MagicStack / MagicStack/uvloop

signal handler sockets error on close leaving loop in 'running' state indefinitely

Open
#625 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cython
Stars
11.9k
Forks
616
PR merge metrics
No merged PRs in 30d

Description

  • uvloop version: 0.19.0 and 0.20.0
  • Python version: 3.12
  • Platform: linux/aws lambda
  • Can you reproduce the bug with PYTHONASYNCIODEBUG in env?:
  • Does uvloop behave differently from vanilla asyncio? How?: have observed the same error on asyncio

I have not been able to come up with a reproduction for this as I think it has something to do with how aws lambda freezes and thaws lambdas.

I have an aws lambda using uvloop that handles requests with using loop.run_until_complete. I can't use uvloop.run or aysncio.run since I need client connected to that loop to still be present on the next invocation of the lambda. Occassionally we have observed errors on the lambda with the following stack trace:

[ERROR] OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
  File "/my_code.py", line 302, in request_handler
    self.loop.run_until_complete(_process_records())
  File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever
  File "uvloop/loop.pyx", line 547, in uvloop.loop.Loop._run
  File "uvloop/loop.pyx", line 348, in uvloop.loop.Loop._pause_signals
  File "/root/.pyenv/versions/3.12.1/lib/python3.12/socket.py", line 504, in close
    self._real_close()
  File "/root/.pyenv/versions/3.12.1/lib/python3.12/socket.py", line 498, in _real_close
    _ss.close(self)

Because this error happens before the loop sets it's self._running field to 0 this means on the next invocation of the lambda the call to loop.run_until_complete errors with a loop is already running error.

I think the bad file descriptor is caused by some feature of the lambda environment or it could just be some kind of random chance since we have a lot of these running and this happens on very small fraction (<0.01%).

I wrote this patch to catch and ignore the Bad file descriptor error, but don't know if that's the right thing to do in the absence of real test case. If anyone has ideas about how I could produce a test case I am happy to try them out.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading uvloop/loop.pyx around Loop._pause_signals and Loop.run_until_complete, then inspect the linked catch-exception-on-pause patch. Reproduce the bad-file-descriptor path, including repeated run_until_complete calls, and add a regression test; done means the loop does not remain marked running after the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.