MagicStack / MagicStack/asyncpg

Client not reconnecting after postgres restarting

Open
#447 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.1k
Forks
468
PR merge metrics
No merged PRs in 30d

Description

* **asyncpg version**: asyncpg==0.18.3
* **PostgreSQL version**: PostgreSQL 10.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit
* **Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?**:
* **Python version**: 3.6
* **Platform**: Ubuntu
* **Do you use pgbouncer?**: no
* **Did you install asyncpg with pip?**: yes
* **If you built asyncpg locally, which version of Cython did you use?**:
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**:

I'm using this library using Tortoise ORM but the trace back is going to the asyncpg library. so I hope you can share and help with this issue.

**Describe the bug**
ORM is not reconnect

**To Reproduce**

1. connect to postgres instance
2. restart postgres
3. try to execute query within the existing python process
4. see exception

**Expected behavior**
1. reconnect when postgres instance is up

**Additional context**
Trace-back1

```
Traceback (most recent call last):
File \"/usr/lib/python3.6/asyncio/selector_events.py\", line 714, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/models.py\", line 277, in create
await instance.save(using_db=kwargs.get('using_db'))
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/models.py\", line 221, in save
await self._insert_instance(*args, **kwargs)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/models.py\", line 210, in _insert_instance
).execute_insert(self)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/backends/base/executor.py\", line 77, in execute_insert
instance.id = await self.db.execute_insert(query, values)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/backends/asyncpg/client.py\", line 22, in wrapped
return await func(self, query, *args)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/tortoise/backends/asyncpg/client.py\", line 108, in execute_insert
stmt = await connection.prepare(query)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/asyncpg/connection.py\", line 403, in prepare
return await self._prepare(query, timeout=timeout, use_cache=False)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/asyncpg/connection.py\", line 408, in _prepare
use_cache=use_cache)
File \"/home/shlomy/PycharmProjects/playground/venv36/lib/python3.6/site-packages/asyncpg/connection.py\", line 329, in _get_statement
statement = await self._protocol.prepare(stmt_name, query, timeout)
File \"asyncpg/protocol/protocol.pyx\", line 163, in prepare
asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation"}
```

Trace back 2
```
connection is closed", "level": "error", "logger": "app.controller", "timestamp": "2019-05-23 12:30:55", "exception": "Traceback (most recent call last):
result = await orm_class.filter(id__in=kwargs['devices_ids']).all()
File \"/usr/local/lib/python3.7/site-packages/tortoise/queryset.py\", line 442, in _execute
self.query, custom_fields=list(self._annotations.keys())
File \"/usr/local/lib/python3.7/site-packages/tortoise/backends/base/executor.py\", line 28, in execute_select
raw_results = await self.db.execute_query(query.get_sql())
File \"/usr/local/lib/python3.7/site-packages/tortoise/backends/asyncpg/client.py\", line 22, in wrapped
return await func(self, query, *args)
File \"/usr/local/lib/python3.7/site-packages/tortoise/backends/asyncpg/client.py\", line 115, in execute_query
return await connection.fetch(query)
File \"/usr/local/lib/python3.7/site-packages/asyncpg/connection.py\", line 420, in fetch
self._check_open()
File \"/usr/local/lib/python3.7/site-packages/asyncpg/connection.py\", line 1147, in _check_open
raise exceptions.InterfaceError('connection is closed')
asyncpg.exceptions._base.InterfaceError: connection is closed"}
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the connection handling shown in asyncpg/connection.py and asyncpg/protocol/protocol.pyx, then reproduce the reported sequence: connect, restart PostgreSQL, and execute a query. Compare the resulting ConnectionDoesNotExistError and InterfaceError with the expected reconnect behavior; done means a query can execute after PostgreSQL is back up.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.