MagicStack / MagicStack/asyncpg

DuplicatePreparedStatementError not solved for Pgbouncer behind ha settings

Aberta
#239 4 comentários 0 reações 0 responsáveis Ver no GitHub

Ninguém assumiu esta issue ainda.

Linguagem predominante
Python
Estrelas
8.1k
Forks
468
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

  • asyncpg version: 0.14.0
  • PostgreSQL version: 9.6.6
  • Python version: 3.6.4
  • Platform: Alpine Linux
  • Do you use pgbouncer?: Yes, in pool_mode=session
  • Did you install asyncpg with pip?: Yes
  • If you built asyncpg locally, which version of Cython did you use?: 0.27.3
  • Can the issue be reproduced under both asyncio and
    uvloop?
    :
    I don't think so

The problem is persisting when a cluster setup (stolon-ha in Kubernetes) is configured. This is the configuration:

  • 5 nodes (1 master, 4 slaves hot replicas)
  • 2 proxies, one to the elected master always, one that distribute reads to all 5 nodes
  • When tried to use asyncpg pool directly to the Kubernetes ClusterIP proxies, they were very unreliable mostly because it does not sanitize, heartbeat and reconnect to those proxies, I had 25% of Connection's errors of all types (Stopped in the middle of operation, connection refused, connection does not exist, etc.)
  • So we put 2 pgbouncer in front *just to keep the pools healthy, with an aggressive configuration for connection checks, sanitation and reconnecting. 99% of connection problems wiped out.
  • Problem come when using cursors (statement_cache=0, pg_bouncer.pool_mode=session) that this problem arises, it does not with a single instance deployed

My guess: proxies (above all round robin slaves) ara a "single connection" that round robins by TCP to all the nodes, so I don't know what's going on but it started to raise this error in the 25% of read queries, what is pretty annoying, all other environments with a single node and a single server pgbouncer works. Should there be a more robust statement name generation so the names have a real random component? that is avoid that 2 processes can clash into the same name (even if in ideal conditions this would mean no problem because you can detect it or whatever). I think random statement names would solve most of the issues, asyncpg_stmt_N_XXXX would avoid that two asyncpg_stmt_8* clashes in these edge cases.

[23:46:06][ERROR] robbie.http.microservice.middleware.errors api.py:__call__:242 | ('Error during SQL operation: prepared statement "__asyncpg_stmt_8__" already exists\nHINT:  \nNOTE: pgbouncer with pool_mode set to "transaction" or\n"statement" does not support prepared statements properly.\nYou have two options:\n\n* if you are using pgbouncer for connection pooling to a\n  single server, switch to the connection pool functionality\n  provided by asyncpg, it is a much better option for this\n  purpose;\n\n* if you have no option of avoiding the use of pgbouncer,\n  then you must switch pgbouncer\'s pool_mode to "session".\n', DuplicatePreparedStatementError('prepared statement "__asyncpg_stmt_8__" already exists',))
Traceback (most recent call last):
  File "robbie/sql/postgresql/utils.py", line 67, in robbie.sql.postgresql.utils.sql_exceptions_handler_asyncgen.wrapper
  File "robbie/sql/postgresql/client.py", line 112, in fetch
  File "robbie/sql/postgresql/client.py", line 113, in robbie.sql.postgresql.client.PgSQLClient.fetch
  File "robbie/sql/postgresql/client.py", line 114, in robbie.sql.postgresql.client.PgSQLClient.fetch
  File "/usr/local/lib/python3.6/site-packages/asyncpg/cursor.py", line 174, in __anext__
    self._query, self._timeout, named=True)
  File "/usr/local/lib/python3.6/site-packages/asyncpg/connection.py", line 286, in _get_statement
    statement = await self._protocol.prepare(stmt_name, query, timeout)
  File "asyncpg/protocol/protocol.pyx", line 168, in prepare
asyncpg.exceptions.DuplicatePreparedStatementError: prepared statement "__asyncpg_stmt_8__" already exists
HINT:
NOTE: pgbouncer with pool_mode set to "transaction" or
"statement" does not support prepared statements properly.
You have two options:

* if you are using pgbouncer for connection pooling to a
  single server, switch to the connection pool functionality
  provided by asyncpg, it is a much better option for this
  purpose;

* if you have no option of avoiding the use of pgbouncer,
  then you must switch pgbouncer's pool_mode to "session".

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Direção de pesquisa

Comece por asyncpg/connection.py em _get_statement e asyncpg/protocol/protocol.pyx em prepare, os pontos de entrada mostrados no traceback. Reproduza o DuplicatePreparedStatementError usando o pool de sessão do pgbouncer e a configuração de HA proxy descritos e, em seguida, verifique se o comportamento foi resolvido sem afetar consultas com cursor ou implantações de nó único.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
postgresql, python
Domínio
backend, databases
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Precisa de esclarecimento
Facilidade para iniciantes
25/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.