tortoise / tortoise/tortoise-orm

transaction task not correctly exited at asyncio.exceptions.CancelledError

Open
#1,152 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5.6k
Forks
516
Avg merge
2d 21h
Merged PRs (30d)
9

Description

Describe the bug
I was using get_or_create as part of my atomic operation, basically tested it with exiting the task real quick. Then I found some errors blocking the server before it shutdown.

To Reproduce
[Code]
here is part of the code in a sanic server, routing to a websocket blueprint

bp_ws = Blueprint("Websockets", url_prefix="/ws")

@bp_ws.websocket("/file/<pk>/")
async def feed(request, ws, pk):
    await Room.get_or_create(channel_name='test_channel')

[Do]

  1. Run server
  2. use postman to connect to the websocket
  3. disconnect in postman real quickly to try exit in the middle of the get_or_create action

[Errors]

Traceback (most recent call last):
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/models.py", line 1057, in get_or_create
    await cls.select_for_update().filter(**kwargs).using_db(connection).get(),
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/queryset.py", line 1006, in _execute
    instance_list = await self._db.executor_class(
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/base/executor.py", line 130, in execute_select
    _, raw_results = await self.db.execute_query(query.get_sql())
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/mysql/client.py", line 44, in translate_exceptions_
    return await func(self, *args)
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/mysql/client.py", line 199, in execute_query
    await cursor.execute(query, values)
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/cursors.py", line 239, in execute
    await self._query(query)
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/cursors.py", line 457, in _query
    await conn.query(q)
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 469, in query
    await self._read_query_result(unbuffered=unbuffered)
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 672, in _read_query_result
    await result.read()
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 1153, in read
    first_packet = await self.connection._read_packet()
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 598, in _read_packet
    packet_header = await self._read_bytes(4)
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 646, in _read_bytes
    data = await self._reader.readexactly(num_bytes)
  File "/home/yuzixin/usr/lib/python3.10/asyncio/streams.py", line 708, in readexactly
    await self._wait_for_data('readexactly')
  File "/home/yuzixin/usr/lib/python3.10/asyncio/streams.py", line 502, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/sanic/app.py", line 994, in _websocket_handler
    await fut
  File "/home/yuzixin/workspace/sanicserver/filesystem/blueprint.py", line 19, in feed
    await Room.get_or_create(channel_name='test_channel')
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/models.py", line 1054, in get_or_create
    async with in_transaction(connection_name=db.connection_name) as connection:
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/base/client.py", line 280, in __aexit__
    await self.connection.rollback()
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/mysql/client.py", line 255, in rollback
    await self._connection.rollback()
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 398, in rollback
    await self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 695, in _execute_command
    self._ensure_alive()
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 1114, in _ensure_alive
    raise InterfaceError(self._close_reason)
pymysql.err.InterfaceError: Cancelled during execution

Error at shutting down the server (a cold shutdown, as the warm shutdown would take forever)

Process ForkProcess-4:
Traceback (most recent call last):
  File "/home/yuzixin/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/yuzixin/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/sanic/server/runners.py", line 191, in serve
    loop.run_until_complete(app._server_event("shutdown", "after"))
  File "uvloop/loop.pyx", line 1499, in uvloop.loop.Loop.run_until_complete
RuntimeError: Event loop stopped before Future completed.

Same error occurs in __aenter__ as well, depending on the timing disconnection is triggered.

I tried adding a simple try except logic to __aexit__ at line 279 and 281, tortoise/backends/base/client.py, simply passing the exception and it is not blocking the server loop anymore. I am not an expert on async programming, but I feel like this might not be a final solution.

I was actually hoping this situation rarely occurs, as it looks like "trying to close mysql connection after ensure_connection but before the next several lines of code". But as I observed, this happened once every two tries

Expected behavior
Correctly closing the task in the middle of a transaction

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tortoise/backends/base/client.py, especially the transaction context manager's aenter and aexit paths shown in the traceback. Reproduce cancellation during Room.get_or_create in the MySQL-backed websocket example, then check that transaction cleanup and server shutdown complete without the rollback or event-loop errors described.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, python
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.