MagicStack / MagicStack/asyncpg

copy_from_query fails without uvloop

オープン
#529 コメント 2 件 リアクション 9 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

主要言語
Python
スター
8.1k
フォーク
468
PR マージ指標
30日以内にマージされた PR はありません

説明

* **asyncpg version**: 0.20.1
* **PostgreSQL version**: PostgreSQL 11.5, compiled by Visual C++ build 1800, 64-bit
* **Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?**: Azure
* **Python version**: 3.8.1
* **Platform**: Windows
* **Do you use pgbouncer?**: No
* **Did you install asyncpg with pip?**: Yes
* **If you built asyncpg locally, which version of Cython did you use?**:
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**: On Linux system work with uvloop but not without uvloop.

Trying to use copy_from_query to make a csv from a view, with 27 000 rows. But it fails on windows or on Linux without uvloop. If I limit the number of rows to 100 it works.

Note: It works fine on Linux with uvloop, but not without uvloop.

Error message on windows:
`Traceback (most recent call last):
File "C:\Program Files\Python38\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python38\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\kf9762\Desktop\spot\project\app\__main__.py", line 46, in
asyncio.run(query())
File "C:\Program Files\Python38\lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 612, in run_until_complete
return future.result()
File "C:\Users\kf9762\Desktop\spot\project\app\__main__.py", line 40, in query
await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv")
File "C:\Users\kf9762\Desktop\spot\project\.venv\lib\site-packages\asyncpg\connection.py", line 591, in copy_from_query
return await self._copy_out(copy_stmt, output, timeout)
File "C:\Users\kf9762\Desktop\spot\project\.venv\lib\site-packages\asyncpg\connection.py", line 796, in _copy_out
return await self._protocol.copy_out(copy_stmt, writer, timeout)
File "asyncpg\protocol\protocol.pyx", line 341, in copy_out
File "asyncpg\protocol\protocol.pyx", line 342, in asyncpg.protocol.protocol.BaseProtocol.copy_out
File "asyncpg\protocol\coreproto.pyx", line 92, in asyncpg.protocol.protocol.CoreProtocol._read_server_messages
File "asyncpg\protocol\coreproto.pyx", line 359, in asyncpg.protocol.protocol.CoreProtocol._process__copy_out_data
File "asyncpg\protocol\coreproto.pyx", line 428, in asyncpg.protocol.protocol.CoreProtocol._parse_copy_data_msgs
File "asyncpg\protocol\protocol.pyx", line 847, in asyncpg.protocol.protocol.BaseProtocol._on_result
File "asyncpg\protocol\protocol.pyx", line 769, in asyncpg.protocol.protocol.BaseProtocol._dispatch_result
AttributeError: 'NoneType' object has no attribute 'cancelled'
Fatal error on SSL transport
protocol:
transport: <_ProactorSocketTransport fd=612 read=<_OverlappedFuture cancelled>>
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 685, in _process_write_backlog
self._transport.write(chunk)
File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 359, in write
self._loop_writing(data=bytes(data))
File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 395, in _loop_writing
self._write_fut = self._loop._proactor.send(self._sock, data)
AttributeError: 'NoneType' object has no attribute 'send'
Exception ignored in:
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 322, in __del__
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 317, in close
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 594, in _start_shutdown
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 599, in _write_appdata
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 707, in _process_write_backlog
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 721, in _fatal_error
File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 151, in _force_close
File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 715, in call_soon
File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 508, in _check_closed
RuntimeError: Event loop is closed`

Error on Linux without uvloop:
`Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/mnt/app/__main__.py", line 49, in
asyncio.run(query())
File "/usr/local/lib/python3.8/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 612, in run_until_complete
return future.result()
File "/mnt/app/__main__.py", line 42, in query
await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv")
File "/usr/local/lib/python3.8/site-packages/asyncpg/connection.py", line 591, in copy_from_query
return await self._copy_out(copy_stmt, output, timeout)
File "/usr/local/lib/python3.8/site-packages/asyncpg/connection.py", line 796, in _copy_out
return await self._protocol.copy_out(copy_stmt, writer, timeout)
File "asyncpg/protocol/protocol.pyx", line 341, in copy_out
File "asyncpg/protocol/protocol.pyx", line 342, in asyncpg.protocol.protocol.BaseProtocol.copy_out
File "asyncpg/protocol/coreproto.pyx", line 92, in asyncpg.protocol.protocol.CoreProtocol._read_server_messages
File "asyncpg/protocol/coreproto.pyx", line 359, in asyncpg.protocol.protocol.CoreProtocol._process__copy_out_data
File "asyncpg/protocol/coreproto.pyx", line 428, in asyncpg.protocol.protocol.CoreProtocol._parse_copy_data_msgs
File "asyncpg/protocol/protocol.pyx", line 847, in asyncpg.protocol.protocol.BaseProtocol._on_result
File "asyncpg/protocol/protocol.pyx", line 769, in asyncpg.protocol.protocol.BaseProtocol._dispatch_result
AttributeError: 'NoneType' object has no attribute 'cancelled'`

Here is the code:
`import asyncio
import asyncpg

async def query():
con = await asyncpg.connect(auth here)
await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv")
await con.close()

if __name__ == '__main__':
asyncio.run(query())`

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

調査の方向性

まず、issue に示されている大きな結果に対する copy_from_query 呼び出しを uvloop なしの asyncio 上で再現し、その後、報告された traceback の周辺にある asyncpg/connection.py、protocol.pyx、coreproto.pyx を調査します。uvloop なしで Windows と Linux 上でコピーが確実に完了し、この失敗に対する回帰テストのカバレッジがあれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
postgresql, python
領域
api, database
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。