MagicStack / MagicStack/asyncpg
Client not reconnecting after postgres restarting
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 468
- PR merge metrics
- No merged PRs in 30d
Description
* **asyncpg version**: asyncpg==0.18.3
* **PostgreSQL version**: PostgreSQL 10.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit
* **Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?**:
* **Python version**: 3.6
* **Platform**: Ubuntu
* **Do you use pgbouncer?**: no
* **Did you install asyncpg with pip?**: yes
* **If you built asyncpg locally, which version of Cython did you use?**:
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**:
I'm using this library using Tortoise ORM but the trace back is going to the asyncpg library. so I hope you can share and help with this issue.
**Describe the bug**
ORM is not reconnect
**To Reproduce**
1. connect to postgres instance
2. restart postgres
3. try to execute query within the existing python process
4. see exception
**Expected behavior**
1. reconnect when postgres instance is up
**Additional context**
Trace-back1
```
Traceback (most recent call last):
File \"/usr/lib/python3.6/asyncio/selector_events.py\", line 714, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/models.py\", line 277, in create
await instance.save(using_db=kwargs.get('using_db'))
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/models.py\", line 221, in save
await self._insert_instance(*args, **kwargs)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/models.py\", line 210, in _insert_instance
).execute_insert(self)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/backends/base/executor.py\", line 77, in execute_insert
instance.id = await self.db.execute_insert(query, values)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/backends/asyncpg/client.py\", line 22, in wrapped
return await func(self, query, *args)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/backends/asyncpg/client.py\", line 108, in execute_insert
stmt = await connection.prepare(query)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/asyncpg/connection.py\", line 403, in prepare
return await self._prepare(query, timeout=timeout, use_cache=False)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/asyncpg/connection.py\", line 408, in _prepare
use_cache=use_cache)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/asyncpg/connection.py\", line 329, in _get_statement
statement = await self._protocol.prepare(stmt_name, query, timeout)
File \"asyncpg/protocol/protocol.pyx\", line 163, in prepare
asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation"}
```
Trace back 2
```
connection is closed", "level": "error", "logger": "app.controller", "timestamp": "2019-05-23 12:30:55", "exception": "Traceback (most recent call last):
result = await orm_class.filter(id__in=kwargs['devices_ids']).all()
File \"/usr/local/lib/python3.7/site-packages/tortoise/queryset.py\", line 442, in _execute
self.query, custom_fields=list(self._annotations.keys())
File \"/usr/local/lib/python3.7/site-packages/tortoise/backends/base/executor.py\", line 28, in execute_select
raw_results = await self.db.execute_query(query.get_sql())
File \"/usr/local/lib/python3.7/site-packages/tortoise/backends/asyncpg/client.py\", line 22, in wrapped
return await func(self, query, *args)
File \"/usr/local/lib/python3.7/site-packages/tortoise/backends/asyncpg/client.py\", line 115, in execute_query
return await connection.fetch(query)
File \"/usr/local/lib/python3.7/site-packages/asyncpg/connection.py\", line 420, in fetch
self._check_open()
File \"/usr/local/lib/python3.7/site-packages/asyncpg/connection.py\", line 1147, in _check_open
raise exceptions.InterfaceError('connection is closed')
asyncpg.exceptions._base.InterfaceError: connection is closed"}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the connection handling shown in asyncpg/connection.py and asyncpg/protocol/protocol.pyx, then reproduce the reported sequence: connect, restart PostgreSQL, and execute a query. Compare the resulting ConnectionDoesNotExistError and InterfaceError with the expected reconnect behavior; done means a query can execute after PostgreSQL is back up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100