MagicStack / MagicStack/asyncpg
DuplicatePreparedStatementError not solved for Pgbouncer behind ha settings
Personne n'a encore pris cette issue.
- Langage dominant
- Python
- Étoiles
- 8.1k
- Forks
- 468
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
- asyncpg version: 0.14.0
- PostgreSQL version: 9.6.6
- Python version: 3.6.4
- Platform: Alpine Linux
- Do you use pgbouncer?: Yes, in pool_mode=session
- Did you install asyncpg with pip?: Yes
- If you built asyncpg locally, which version of Cython did you use?: 0.27.3
- Can the issue be reproduced under both asyncio and
uvloop?:
I don't think so
The problem is persisting when a cluster setup (stolon-ha in Kubernetes) is configured. This is the configuration:
- 5 nodes (1 master, 4 slaves hot replicas)
- 2 proxies, one to the elected master always, one that distribute reads to all 5 nodes
- When tried to use asyncpg pool directly to the Kubernetes ClusterIP proxies, they were very unreliable mostly because it does not sanitize, heartbeat and reconnect to those proxies, I had 25% of Connection's errors of all types (Stopped in the middle of operation, connection refused, connection does not exist, etc.)
- So we put 2 pgbouncer in front *just to keep the pools healthy, with an aggressive configuration for connection checks, sanitation and reconnecting. 99% of connection problems wiped out.
- Problem come when using cursors (statement_cache=0, pg_bouncer.pool_mode=session) that this problem arises, it does not with a single instance deployed
My guess: proxies (above all round robin slaves) ara a "single connection" that round robins by TCP to all the nodes, so I don't know what's going on but it started to raise this error in the 25% of read queries, what is pretty annoying, all other environments with a single node and a single server pgbouncer works. Should there be a more robust statement name generation so the names have a real random component? that is avoid that 2 processes can clash into the same name (even if in ideal conditions this would mean no problem because you can detect it or whatever). I think random statement names would solve most of the issues, asyncpg_stmt_N_XXXX would avoid that two asyncpg_stmt_8* clashes in these edge cases.
[23:46:06][ERROR] robbie.http.microservice.middleware.errors api.py:__call__:242 | ('Error during SQL operation: prepared statement "__asyncpg_stmt_8__" already exists\nHINT: \nNOTE: pgbouncer with pool_mode set to "transaction" or\n"statement" does not support prepared statements properly.\nYou have two options:\n\n* if you are using pgbouncer for connection pooling to a\n single server, switch to the connection pool functionality\n provided by asyncpg, it is a much better option for this\n purpose;\n\n* if you have no option of avoiding the use of pgbouncer,\n then you must switch pgbouncer\'s pool_mode to "session".\n', DuplicatePreparedStatementError('prepared statement "__asyncpg_stmt_8__" already exists',))
Traceback (most recent call last):
File "robbie/sql/postgresql/utils.py", line 67, in robbie.sql.postgresql.utils.sql_exceptions_handler_asyncgen.wrapper
File "robbie/sql/postgresql/client.py", line 112, in fetch
File "robbie/sql/postgresql/client.py", line 113, in robbie.sql.postgresql.client.PgSQLClient.fetch
File "robbie/sql/postgresql/client.py", line 114, in robbie.sql.postgresql.client.PgSQLClient.fetch
File "/usr/local/lib/python3.6/site-packages/asyncpg/cursor.py", line 174, in __anext__
self._query, self._timeout, named=True)
File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 286, in _get_statement
statement = await self._protocol.prepare(stmt_name, query, timeout)
File "asyncpg/protocol/protocol.pyx", line 168, in prepare
asyncpg.exceptions.DuplicatePreparedStatementError: prepared statement "__asyncpg_stmt_8__" already exists
HINT:
NOTE: pgbouncer with pool_mode set to "transaction" or
"statement" does not support prepared statements properly.
You have two options:
* if you are using pgbouncer for connection pooling to a
single server, switch to the connection pool functionality
provided by asyncpg, it is a much better option for this
purpose;
* if you have no option of avoiding the use of pgbouncer,
then you must switch pgbouncer's pool_mode to "session".
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Piste de recherche
Commencez par asyncpg/connection.py à _get_statement et asyncpg/protocol/protocol.pyx à prepare, les points d’entrée indiqués dans la trace d’erreur. Reproduisez le DuplicatePreparedStatementError avec le pool de session pgbouncer et la configuration HA proxy décrits, puis vérifiez que le comportement est résolu sans affecter les requêtes avec curseur ni les déploiements sur un seul nœud.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- postgresql, python
- Domaine
- backend, databases
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 25/100