MagicStack / MagicStack/asyncpg
DuplicatePreparedStatementError not solved for Pgbouncer behind ha settings
Chưa có ai nhận issue này.
- Ngôn ngữ chính
- Python
- Star
- 8.1k
- Fork
- 468
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
* **asyncpg version**: 0.14.0
* **PostgreSQL version**: 9.6.6
* **Python version**: 3.6.4
* **Platform**: Alpine Linux
* **Do you use pgbouncer?**: Yes, in pool_mode=session
* **Did you install asyncpg with pip?**: Yes
* **If you built asyncpg locally, which version of Cython did you use?**: 0.27.3
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**:
I don't think so
The problem is persisting when a cluster setup (stolon-ha in Kubernetes) is configured. This is the configuration:
- 5 nodes (1 master, 4 slaves hot replicas)
- 2 proxies, one to the elected master always, one that distribute reads to all 5 nodes
- When tried to use asyncpg pool directly to the Kubernetes ClusterIP proxies, they were very unreliable mostly because it does not sanitize, heartbeat and reconnect to those proxies, I had 25% of Connection's errors of all types (Stopped in the middle of operation, connection refused, connection does not exist, etc.)
- So we put 2 pgbouncer in front *just to keep the pools healthy, with an aggressive configuration for connection checks, sanitation and reconnecting. 99% of connection problems wiped out.
- Problem come when using cursors (statement_cache=0, pg_bouncer.pool_mode=session) that this problem arises, it does not with a single instance deployed
My guess: proxies (above all round robin slaves) ara a "single connection" that round robins by TCP to all the nodes, so I don't know what's going on but it started to raise this error in the 25% of read queries, what is pretty annoying, all other environments with a single node and a single server pgbouncer works. Should there be a more robust statement name generation so the names have a real random component? that is avoid that 2 processes can clash into the same name (even if in ideal conditions this would mean no problem because you can detect it or whatever). I think random statement names would solve most of the issues, __asyncpg_stmt_N_XXXX__ would avoid that two __asyncpg_stmt_8__* clashes in these edge cases.
```
[23:46:06][ERROR] robbie.http.microservice.middleware.errors api.py:__call__:242 | ('Error during SQL operation: prepared statement "__asyncpg_stmt_8__" already exists\nHINT: \nNOTE: pgbouncer with pool_mode set to "transaction" or\n"statement" does not support prepared statements properly.\nYou have two options:\n\n* if you are using pgbouncer for connection pooling to a\n single server, switch to the connection pool functionality\n provided by asyncpg, it is a much better option for this\n purpose;\n\n* if you have no option of avoiding the use of pgbouncer,\n then you must switch pgbouncer\'s pool_mode to "session".\n', DuplicatePreparedStatementError('prepared statement "__asyncpg_stmt_8__" already exists',))
Traceback (most recent call last):
File "robbie/sql/postgresql/utils.py", line 67, in robbie.sql.postgresql.utils.sql_exceptions_handler_asyncgen.wrapper
File "robbie/sql/postgresql/client.py", line 112, in fetch
File "robbie/sql/postgresql/client.py", line 113, in robbie.sql.postgresql.client.PgSQLClient.fetch
File "robbie/sql/postgresql/client.py", line 114, in robbie.sql.postgresql.client.PgSQLClient.fetch
File "/usr/local/lib/python3.6/site-packages/asyncpg/cursor.py", line 174, in __anext__
self._query, self._timeout, named=True)
File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 286, in _get_statement
statement = await self._protocol.prepare(stmt_name, query, timeout)
File "asyncpg/protocol/protocol.pyx", line 168, in prepare
asyncpg.exceptions.DuplicatePreparedStatementError: prepared statement "__asyncpg_stmt_8__" already exists
HINT:
NOTE: pgbouncer with pool_mode set to "transaction" or
"statement" does not support prepared statements properly.
You have two options:
* if you are using pgbouncer for connection pooling to a
single server, switch to the connection pool functionality
provided by asyncpg, it is a much better option for this
purpose;
* if you have no option of avoiding the use of pgbouncer,
then you must switch pgbouncer's pool_mode to "session".
```
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Hướng nghiên cứu
Bắt đầu với asyncpg/connection.py tại _get_statement và asyncpg/protocol/protocol.pyx tại prepare, các điểm vào được hiển thị trong traceback. Tái hiện DuplicatePreparedStatementError bằng cách sử dụng session pool của pgbouncer và cấu hình HA proxy đã mô tả, sau đó xác minh rằng hành vi này đã được khắc phục mà không ảnh hưởng đến các truy vấn cursor hoặc các triển khai một nút.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- postgresql, python
- Lĩnh vực
- backend, databases
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Cần làm rõ
- Mức phù hợp với người mới
- 25/100