danielgtaylor / danielgtaylor/python-betterproto

AsyncChannel produces unexpected exception on cancel

Open
#233 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.8k
Forks
234
PR merge metrics
No merged PRs in 30d

Description

Hey there,

I'm using the AsyncChannel to send messages to a GRPC server. When receiving a grpc error the client prints some non-catched errors of the sending task in the console (related to: https://github.com/danielgtaylor/python-betterproto/issues/188).

future: <Task finished name='Task-2' coro=<ServiceStub._send_messages() done, defined at /home/PycharmProjects/Worker/venv/lib/python3.8/site-packages/betterproto/grpc/grpclib_client.py:163> exception=ValueError('task_done() called too many times')>
Traceback (most recent call last):
  File "/home/PycharmProjects/Worker/venv/lib/python3.8/site-packages/betterproto/grpc/util/async_channel.py", line 87, in __anext__
    result = await self._queue.get()
  File "/usr/lib/python3.8/asyncio/queues.py", line 163, in get
    await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/PycharmProjects/Worker/venv/lib/python3.8/site-packages/betterproto/grpc/grpclib_client.py", line 166, in _send_messages
    async for message in messages:
  File "/home/PycharmProjects/Worker/venv/lib/python3.8/site-packages/betterproto/grpc/util/async_channel.py", line 93, in __anext__
    self._queue.task_done()
  File "/usr/lib/python3.8/asyncio/queues.py", line 206, in task_done
    raise ValueError('task_done() called too many times')
ValueError: task_done() called too many times

When the server sends a grpc-error the error is handled in this except-block, which cancels the sending task:
https://github.com/danielgtaylor/python-betterproto/blob/master/src/betterproto/grpc/grpclib_client.py#L159-L161

This cancel-method results in throwing a CancelledError inside the current await of the sending task, which is the get-method on the queue of the AsyncChannel:
https://github.com/danielgtaylor/python-betterproto/blob/master/src/betterproto/grpc/util/async_channel.py#L87

In this case the get-method is raising a CancelledError even if the queue is empty. If this happens the task_done-method is called on an empty queue, and the above error message is shown.

A quick fix would be the following, however I haven't checked if this fix has any side effects:

try:
    ...
finally:
    self._waiting_receivers -= 1
    if not self._queue.empty():
        self._queue.task_done()

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/betterproto/grpc/util/async_channel.py, especially anext around the queue get and task_done calls, then read src/betterproto/grpc/grpclib_client.py around the cancellation of _send_messages. Reproduce cancellation while the queue is empty and verify that the sending task no longer reports a task_done() called too many times exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.