MagicStack / MagicStack/asyncpg

copy_from_query fails without uvloop

未关闭
#529 2 条评论 9 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

主要语言
Python
星标
8.1k
派生
468
PR 合并指标
30 天内没有已合并 PR

描述

  • asyncpg version: 0.20.1
  • PostgreSQL version: PostgreSQL 11.5, compiled by Visual C++ build 1800, 64-bit
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : Azure
  • Python version: 3.8.1
  • Platform: Windows
  • Do you use pgbouncer?: No
  • Did you install asyncpg with pip?: Yes
  • If you built asyncpg locally, which version of Cython did you use?:
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : On Linux system work with uvloop but not without uvloop.

Trying to use copy_from_query to make a csv from a view, with 27 000 rows. But it fails on windows or on Linux without uvloop. If I limit the number of rows to 100 it works.

Note: It works fine on Linux with uvloop, but not without uvloop.

Error message on windows:
Traceback (most recent call last): File "C:\Program Files\Python38\lib\runpy.py", line 193, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python38\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\kf9762\Desktop\spot\project\app\__main__.py", line 46, in <module> asyncio.run(query()) File "C:\Program Files\Python38\lib\asyncio\runners.py", line 43, in run return loop.run_until_complete(main) File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 612, in run_until_complete return future.result() File "C:\Users\kf9762\Desktop\spot\project\app\__main__.py", line 40, in query await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv") File "C:\Users\kf9762\Desktop\spot\project\.venv\lib\site-packages\asyncpg\connection.py", line 591, in copy_from_query return await self._copy_out(copy_stmt, output, timeout) File "C:\Users\kf9762\Desktop\spot\project\.venv\lib\site-packages\asyncpg\connection.py", line 796, in _copy_out return await self._protocol.copy_out(copy_stmt, writer, timeout) File "asyncpg\protocol\protocol.pyx", line 341, in copy_out File "asyncpg\protocol\protocol.pyx", line 342, in asyncpg.protocol.protocol.BaseProtocol.copy_out File "asyncpg\protocol\coreproto.pyx", line 92, in asyncpg.protocol.protocol.CoreProtocol._read_server_messages File "asyncpg\protocol\coreproto.pyx", line 359, in asyncpg.protocol.protocol.CoreProtocol._process__copy_out_data File "asyncpg\protocol\coreproto.pyx", line 428, in asyncpg.protocol.protocol.CoreProtocol._parse_copy_data_msgs File "asyncpg\protocol\protocol.pyx", line 847, in asyncpg.protocol.protocol.BaseProtocol._on_result File "asyncpg\protocol\protocol.pyx", line 769, in asyncpg.protocol.protocol.BaseProtocol._dispatch_result AttributeError: 'NoneType' object has no attribute 'cancelled' Fatal error on SSL transport protocol: <asyncio.sslproto.SSLProtocol object at 0x00000279CB926A90> transport: <_ProactorSocketTransport fd=612 read=<_OverlappedFuture cancelled>> Traceback (most recent call last): File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 685, in _process_write_backlog self._transport.write(chunk) File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 359, in write self._loop_writing(data=bytes(data)) File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 395, in _loop_writing self._write_fut = self._loop._proactor.send(self._sock, data) AttributeError: 'NoneType' object has no attribute 'send' Exception ignored in: <function _SSLProtocolTransport.__del__ at 0x00000279CB52E940> Traceback (most recent call last): File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 322, in __del__ File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 317, in close File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 594, in _start_shutdown File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 599, in _write_appdata File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 707, in _process_write_backlog File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 721, in _fatal_error File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 151, in _force_close File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 715, in call_soon File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 508, in _check_closed RuntimeError: Event loop is closed

Error on Linux without uvloop:
Traceback (most recent call last): File "/usr/local/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code exec(code, run_globals) File "/mnt/app/__main__.py", line 49, in <module> asyncio.run(query()) File "/usr/local/lib/python3.8/asyncio/runners.py", line 43, in run return loop.run_until_complete(main) File "/usr/local/lib/python3.8/asyncio/base_events.py", line 612, in run_until_complete return future.result() File "/mnt/app/__main__.py", line 42, in query await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv") File "/usr/local/lib/python3.8/site-packages/asyncpg/connection.py", line 591, in copy_from_query return await self._copy_out(copy_stmt, output, timeout) File "/usr/local/lib/python3.8/site-packages/asyncpg/connection.py", line 796, in _copy_out return await self._protocol.copy_out(copy_stmt, writer, timeout) File "asyncpg/protocol/protocol.pyx", line 341, in copy_out File "asyncpg/protocol/protocol.pyx", line 342, in asyncpg.protocol.protocol.BaseProtocol.copy_out File "asyncpg/protocol/coreproto.pyx", line 92, in asyncpg.protocol.protocol.CoreProtocol._read_server_messages File "asyncpg/protocol/coreproto.pyx", line 359, in asyncpg.protocol.protocol.CoreProtocol._process__copy_out_data File "asyncpg/protocol/coreproto.pyx", line 428, in asyncpg.protocol.protocol.CoreProtocol._parse_copy_data_msgs File "asyncpg/protocol/protocol.pyx", line 847, in asyncpg.protocol.protocol.BaseProtocol._on_result File "asyncpg/protocol/protocol.pyx", line 769, in asyncpg.protocol.protocol.BaseProtocol._dispatch_result AttributeError: 'NoneType' object has no attribute 'cancelled'

Here is the code:
`import asyncio
import asyncpg

async def query():
con = await asyncpg.connect(auth here)
await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv")
await con.close()

if name == 'main':
asyncio.run(query())`

贡献指南

这个仓库没有索引到贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先,在不使用 uvloop 的 asyncio 下复现 issue 中所示的大结果 copy_from_query 调用,然后检查报告的 traceback 附近的 asyncpg/connection.py、protocol.pyx 和 coreproto.pyx。完成的标准是:在不使用 uvloop 的情况下,复制操作能够在 Windows 和 Linux 上可靠完成,并且有针对该故障的回归测试覆盖。

由索引模型根据 Issue 内容生成。

评估

技术栈
postgresql, python
领域
api, database
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。