tortoise / tortoise/tortoise-orm
transaction task not correctly exited at asyncio.exceptions.CancelledError
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 516
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 9
Description
Describe the bug
I was using get_or_create as part of my atomic operation, basically tested it with exiting the task real quick. Then I found some errors blocking the server before it shutdown.
To Reproduce
[Code]
here is part of the code in a sanic server, routing to a websocket blueprint
bp_ws = Blueprint("Websockets", url_prefix="/ws")
@bp_ws.websocket("/file/<pk>/")
async def feed(request, ws, pk):
await Room.get_or_create(channel_name='test_channel')
[Do]
- Run server
- use postman to connect to the websocket
- disconnect in postman real quickly to try exit in the middle of the get_or_create action
[Errors]
Traceback (most recent call last):
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/models.py", line 1057, in get_or_create
await cls.select_for_update().filter(**kwargs).using_db(connection).get(),
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/queryset.py", line 1006, in _execute
instance_list = await self._db.executor_class(
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/base/executor.py", line 130, in execute_select
_, raw_results = await self.db.execute_query(query.get_sql())
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/mysql/client.py", line 44, in translate_exceptions_
return await func(self, *args)
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/mysql/client.py", line 199, in execute_query
await cursor.execute(query, values)
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/cursors.py", line 239, in execute
await self._query(query)
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/cursors.py", line 457, in _query
await conn.query(q)
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 469, in query
await self._read_query_result(unbuffered=unbuffered)
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 672, in _read_query_result
await result.read()
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 1153, in read
first_packet = await self.connection._read_packet()
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 598, in _read_packet
packet_header = await self._read_bytes(4)
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 646, in _read_bytes
data = await self._reader.readexactly(num_bytes)
File "/home/yuzixin/usr/lib/python3.10/asyncio/streams.py", line 708, in readexactly
await self._wait_for_data('readexactly')
File "/home/yuzixin/usr/lib/python3.10/asyncio/streams.py", line 502, in _wait_for_data
await self._waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/sanic/app.py", line 994, in _websocket_handler
await fut
File "/home/yuzixin/workspace/sanicserver/filesystem/blueprint.py", line 19, in feed
await Room.get_or_create(channel_name='test_channel')
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/models.py", line 1054, in get_or_create
async with in_transaction(connection_name=db.connection_name) as connection:
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/base/client.py", line 280, in __aexit__
await self.connection.rollback()
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/tortoise/backends/mysql/client.py", line 255, in rollback
await self._connection.rollback()
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 398, in rollback
await self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 695, in _execute_command
self._ensure_alive()
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/aiomysql/connection.py", line 1114, in _ensure_alive
raise InterfaceError(self._close_reason)
pymysql.err.InterfaceError: Cancelled during execution
Error at shutting down the server (a cold shutdown, as the warm shutdown would take forever)
Process ForkProcess-4:
Traceback (most recent call last):
File "/home/yuzixin/usr/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/yuzixin/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/yuzixin/workspace/sanicserver/venv/lib/python3.10/site-packages/sanic/server/runners.py", line 191, in serve
loop.run_until_complete(app._server_event("shutdown", "after"))
File "uvloop/loop.pyx", line 1499, in uvloop.loop.Loop.run_until_complete
RuntimeError: Event loop stopped before Future completed.
Same error occurs in __aenter__ as well, depending on the timing disconnection is triggered.
I tried adding a simple try except logic to __aexit__ at line 279 and 281, tortoise/backends/base/client.py, simply passing the exception and it is not blocking the server loop anymore. I am not an expert on async programming, but I feel like this might not be a final solution.
I was actually hoping this situation rarely occurs, as it looks like "trying to close mysql connection after ensure_connection but before the next several lines of code". But as I observed, this happened once every two tries
Expected behavior
Correctly closing the task in the middle of a transaction
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tortoise/backends/base/client.py, especially the transaction context manager's aenter and aexit paths shown in the traceback. Reproduce cancellation during Room.get_or_create in the MySQL-backed websocket example, then check that transaction cleanup and server shutdown complete without the rollback or event-loop errors described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, python
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100