MagicStack / MagicStack/asyncpg
DuplicatePreparedStatementError not solved for Pgbouncer behind ha settings
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 8.1k
- 派生
- 468
- PR 合并指标
- 30 天内没有已合并 PR
描述
* **asyncpg version**: 0.14.0
* **PostgreSQL version**: 9.6.6
* **Python version**: 3.6.4
* **Platform**: Alpine Linux
* **Do you use pgbouncer?**: Yes, in pool_mode=session
* **Did you install asyncpg with pip?**: Yes
* **If you built asyncpg locally, which version of Cython did you use?**: 0.27.3
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**:
I don't think so
The problem is persisting when a cluster setup (stolon-ha in Kubernetes) is configured. This is the configuration:
- 5 nodes (1 master, 4 slaves hot replicas)
- 2 proxies, one to the elected master always, one that distribute reads to all 5 nodes
- When tried to use asyncpg pool directly to the Kubernetes ClusterIP proxies, they were very unreliable mostly because it does not sanitize, heartbeat and reconnect to those proxies, I had 25% of Connection's errors of all types (Stopped in the middle of operation, connection refused, connection does not exist, etc.)
- So we put 2 pgbouncer in front *just to keep the pools healthy, with an aggressive configuration for connection checks, sanitation and reconnecting. 99% of connection problems wiped out.
- Problem come when using cursors (statement_cache=0, pg_bouncer.pool_mode=session) that this problem arises, it does not with a single instance deployed
My guess: proxies (above all round robin slaves) ara a "single connection" that round robins by TCP to all the nodes, so I don't know what's going on but it started to raise this error in the 25% of read queries, what is pretty annoying, all other environments with a single node and a single server pgbouncer works. Should there be a more robust statement name generation so the names have a real random component? that is avoid that 2 processes can clash into the same name (even if in ideal conditions this would mean no problem because you can detect it or whatever). I think random statement names would solve most of the issues, __asyncpg_stmt_N_XXXX__ would avoid that two __asyncpg_stmt_8__* clashes in these edge cases.
```
[23:46:06][ERROR] robbie.http.microservice.middleware.errors api.py:__call__:242 | ('Error during SQL operation: prepared statement "__asyncpg_stmt_8__" already exists\nHINT: \nNOTE: pgbouncer with pool_mode set to "transaction" or\n"statement" does not support prepared statements properly.\nYou have two options:\n\n* if you are using pgbouncer for connection pooling to a\n single server, switch to the connection pool functionality\n provided by asyncpg, it is a much better option for this\n purpose;\n\n* if you have no option of avoiding the use of pgbouncer,\n then you must switch pgbouncer\'s pool_mode to "session".\n', DuplicatePreparedStatementError('prepared statement "__asyncpg_stmt_8__" already exists',))
Traceback (most recent call last):
File "robbie/sql/postgresql/utils.py", line 67, in robbie.sql.postgresql.utils.sql_exceptions_handler_asyncgen.wrapper
File "robbie/sql/postgresql/client.py", line 112, in fetch
File "robbie/sql/postgresql/client.py", line 113, in robbie.sql.postgresql.client.PgSQLClient.fetch
File "robbie/sql/postgresql/client.py", line 114, in robbie.sql.postgresql.client.PgSQLClient.fetch
File "/usr/local/lib/python3.6/site-packages/asyncpg/cursor.py", line 174, in __anext__
self._query, self._timeout, named=True)
File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 286, in _get_statement
statement = await self._protocol.prepare(stmt_name, query, timeout)
File "asyncpg/protocol/protocol.pyx", line 168, in prepare
asyncpg.exceptions.DuplicatePreparedStatementError: prepared statement "__asyncpg_stmt_8__" already exists
HINT:
NOTE: pgbouncer with pool_mode set to "transaction" or
"statement" does not support prepared statements properly.
You have two options:
* if you are using pgbouncer for connection pooling to a
single server, switch to the connection pool functionality
provided by asyncpg, it is a much better option for this
purpose;
* if you have no option of avoiding the use of pgbouncer,
then you must switch pgbouncer's pool_mode to "session".
```
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
从 traceback 中显示的入口点开始,检查 asyncpg/connection.py 中的 _get_statement 和 asyncpg/protocol/protocol.pyx 中的 prepare。使用所描述的 pgbouncer 会话池和 HA proxy 设置重现 DuplicatePreparedStatementError,然后验证该行为已得到解决,且不会影响游标查询或单节点部署。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- postgresql, python
- 领域
- backend, databases
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100