MagicStack / MagicStack/asyncpg

DuplicatePreparedStatementError not solved for Pgbouncer behind ha settings

Ouverte
#239 4 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Langage dominant
Python
Étoiles
8.1k
Forks
468
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

  • asyncpg version: 0.14.0
  • PostgreSQL version: 9.6.6
  • Python version: 3.6.4
  • Platform: Alpine Linux
  • Do you use pgbouncer?: Yes, in pool_mode=session
  • Did you install asyncpg with pip?: Yes
  • If you built asyncpg locally, which version of Cython did you use?: 0.27.3
  • Can the issue be reproduced under both asyncio and
    uvloop?
    :
    I don't think so

The problem is persisting when a cluster setup (stolon-ha in Kubernetes) is configured. This is the configuration:

  • 5 nodes (1 master, 4 slaves hot replicas)
  • 2 proxies, one to the elected master always, one that distribute reads to all 5 nodes
  • When tried to use asyncpg pool directly to the Kubernetes ClusterIP proxies, they were very unreliable mostly because it does not sanitize, heartbeat and reconnect to those proxies, I had 25% of Connection's errors of all types (Stopped in the middle of operation, connection refused, connection does not exist, etc.)
  • So we put 2 pgbouncer in front *just to keep the pools healthy, with an aggressive configuration for connection checks, sanitation and reconnecting. 99% of connection problems wiped out.
  • Problem come when using cursors (statement_cache=0, pg_bouncer.pool_mode=session) that this problem arises, it does not with a single instance deployed

My guess: proxies (above all round robin slaves) ara a "single connection" that round robins by TCP to all the nodes, so I don't know what's going on but it started to raise this error in the 25% of read queries, what is pretty annoying, all other environments with a single node and a single server pgbouncer works. Should there be a more robust statement name generation so the names have a real random component? that is avoid that 2 processes can clash into the same name (even if in ideal conditions this would mean no problem because you can detect it or whatever). I think random statement names would solve most of the issues, asyncpg_stmt_N_XXXX would avoid that two asyncpg_stmt_8* clashes in these edge cases.

[23:46:06][ERROR] robbie.http.microservice.middleware.errors api.py:__call__:242 | ('Error during SQL operation: prepared statement "__asyncpg_stmt_8__" already exists\nHINT:  \nNOTE: pgbouncer with pool_mode set to "transaction" or\n"statement" does not support prepared statements properly.\nYou have two options:\n\n* if you are using pgbouncer for connection pooling to a\n  single server, switch to the connection pool functionality\n  provided by asyncpg, it is a much better option for this\n  purpose;\n\n* if you have no option of avoiding the use of pgbouncer,\n  then you must switch pgbouncer\'s pool_mode to "session".\n', DuplicatePreparedStatementError('prepared statement "__asyncpg_stmt_8__" already exists',))
Traceback (most recent call last):
  File "robbie/sql/postgresql/utils.py", line 67, in robbie.sql.postgresql.utils.sql_exceptions_handler_asyncgen.wrapper
  File "robbie/sql/postgresql/client.py", line 112, in fetch
  File "robbie/sql/postgresql/client.py", line 113, in robbie.sql.postgresql.client.PgSQLClient.fetch
  File "robbie/sql/postgresql/client.py", line 114, in robbie.sql.postgresql.client.PgSQLClient.fetch
  File "/usr/local/lib/python3.6/site-packages/asyncpg/cursor.py", line 174, in __anext__
    self._query, self._timeout, named=True)
  File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 286, in _get_statement
    statement = await self._protocol.prepare(stmt_name, query, timeout)
  File "asyncpg/protocol/protocol.pyx", line 168, in prepare
asyncpg.exceptions.DuplicatePreparedStatementError: prepared statement "__asyncpg_stmt_8__" already exists
HINT:
NOTE: pgbouncer with pool_mode set to "transaction" or
"statement" does not support prepared statements properly.
You have two options:

* if you are using pgbouncer for connection pooling to a
  single server, switch to the connection pool functionality
  provided by asyncpg, it is a much better option for this
  purpose;

* if you have no option of avoiding the use of pgbouncer,
  then you must switch pgbouncer's pool_mode to "session".

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez par asyncpg/connection.py à _get_statement et asyncpg/protocol/protocol.pyx à prepare, les points d’entrée indiqués dans la trace d’erreur. Reproduisez le DuplicatePreparedStatementError avec le pool de session pgbouncer et la configuration HA proxy décrits, puis vérifiez que le comportement est résolu sans affecter les requêtes avec curseur ni les déploiements sur un seul nœud.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
postgresql, python
Domaine
backend, databases
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
À clarifier
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.