MagicStack / MagicStack/asyncpg
copy_from_query fails without uvloop
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 468
- PR merge metrics
- No merged PRs in 30d
Description
* **asyncpg version**: 0.20.1
* **PostgreSQL version**: PostgreSQL 11.5, compiled by Visual C++ build 1800, 64-bit
* **Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?**: Azure
* **Python version**: 3.8.1
* **Platform**: Windows
* **Do you use pgbouncer?**: No
* **Did you install asyncpg with pip?**: Yes
* **If you built asyncpg locally, which version of Cython did you use?**:
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**: On Linux system work with uvloop but not without uvloop.
Trying to use copy_from_query to make a csv from a view, with 27 000 rows. But it fails on windows or on Linux without uvloop. If I limit the number of rows to 100 it works.
Note: It works fine on Linux with uvloop, but not without uvloop.
Error message on windows:
`Traceback (most recent call last):
File "C:\Program Files\Python38\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python38\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\kf9762\Desktop\spot\project\app\__main__.py", line 46, in
asyncio.run(query())
File "C:\Program Files\Python38\lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 612, in run_until_complete
return future.result()
File "C:\Users\kf9762\Desktop\spot\project\app\__main__.py", line 40, in query
await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv")
File "C:\Users\kf9762\Desktop\spot\project\.venv\lib\site-packages\asyncpg\connection.py", line 591, in copy_from_query
return await self._copy_out(copy_stmt, output, timeout)
File "C:\Users\kf9762\Desktop\spot\project\.venv\lib\site-packages\asyncpg\connection.py", line 796, in _copy_out
return await self._protocol.copy_out(copy_stmt, writer, timeout)
File "asyncpg\protocol\protocol.pyx", line 341, in copy_out
File "asyncpg\protocol\protocol.pyx", line 342, in asyncpg.protocol.protocol.BaseProtocol.copy_out
File "asyncpg\protocol\coreproto.pyx", line 92, in asyncpg.protocol.protocol.CoreProtocol._read_server_messages
File "asyncpg\protocol\coreproto.pyx", line 359, in asyncpg.protocol.protocol.CoreProtocol._process__copy_out_data
File "asyncpg\protocol\coreproto.pyx", line 428, in asyncpg.protocol.protocol.CoreProtocol._parse_copy_data_msgs
File "asyncpg\protocol\protocol.pyx", line 847, in asyncpg.protocol.protocol.BaseProtocol._on_result
File "asyncpg\protocol\protocol.pyx", line 769, in asyncpg.protocol.protocol.BaseProtocol._dispatch_result
AttributeError: 'NoneType' object has no attribute 'cancelled'
Fatal error on SSL transport
protocol:
transport: <_ProactorSocketTransport fd=612 read=<_OverlappedFuture cancelled>>
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 685, in _process_write_backlog
self._transport.write(chunk)
File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 359, in write
self._loop_writing(data=bytes(data))
File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 395, in _loop_writing
self._write_fut = self._loop._proactor.send(self._sock, data)
AttributeError: 'NoneType' object has no attribute 'send'
Exception ignored in:
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 322, in __del__
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 317, in close
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 594, in _start_shutdown
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 599, in _write_appdata
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 707, in _process_write_backlog
File "C:\Program Files\Python38\lib\asyncio\sslproto.py", line 721, in _fatal_error
File "C:\Program Files\Python38\lib\asyncio\proactor_events.py", line 151, in _force_close
File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 715, in call_soon
File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 508, in _check_closed
RuntimeError: Event loop is closed`
Error on Linux without uvloop:
`Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/mnt/app/__main__.py", line 49, in
asyncio.run(query())
File "/usr/local/lib/python3.8/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 612, in run_until_complete
return future.result()
File "/mnt/app/__main__.py", line 42, in query
await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv")
File "/usr/local/lib/python3.8/site-packages/asyncpg/connection.py", line 591, in copy_from_query
return await self._copy_out(copy_stmt, output, timeout)
File "/usr/local/lib/python3.8/site-packages/asyncpg/connection.py", line 796, in _copy_out
return await self._protocol.copy_out(copy_stmt, writer, timeout)
File "asyncpg/protocol/protocol.pyx", line 341, in copy_out
File "asyncpg/protocol/protocol.pyx", line 342, in asyncpg.protocol.protocol.BaseProtocol.copy_out
File "asyncpg/protocol/coreproto.pyx", line 92, in asyncpg.protocol.protocol.CoreProtocol._read_server_messages
File "asyncpg/protocol/coreproto.pyx", line 359, in asyncpg.protocol.protocol.CoreProtocol._process__copy_out_data
File "asyncpg/protocol/coreproto.pyx", line 428, in asyncpg.protocol.protocol.CoreProtocol._parse_copy_data_msgs
File "asyncpg/protocol/protocol.pyx", line 847, in asyncpg.protocol.protocol.BaseProtocol._on_result
File "asyncpg/protocol/protocol.pyx", line 769, in asyncpg.protocol.protocol.BaseProtocol._dispatch_result
AttributeError: 'NoneType' object has no attribute 'cancelled'`
Here is the code:
`import asyncio
import asyncpg
async def query():
con = await asyncpg.connect(auth here)
await con.copy_from_query("SELECT * FROM spot.csv_utc", format='csv', header=True, output="spot.csv")
await con.close()
if __name__ == '__main__':
asyncio.run(query())`
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the large-result copy_from_query call shown in the issue under asyncio without uvloop, then inspect asyncpg/connection.py, protocol.pyx, and coreproto.pyx around the reported traceback. Done means the copy completes reliably on Windows and Linux without uvloop, with regression coverage for the failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- api, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100