MagicStack / MagicStack/asyncpg

Infinite hang on leaving conn.transaction() block

Open
#442 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.1k
Forks
468
PR merge metrics
No merged PRs in 30d

Description

Hi, I am experiencing hangs at aexit of Transaction context manager, with traceback as follows:

...
  File "/usr/local/lib/python3.7/dist-packages/asyncpg/transaction.py", line 89, in __aexit__
    await self.__rollback()
  File "/usr/local/lib/python3.7/dist-packages/asyncpg/transaction.py", line 198, in __rollback
    await self._connection.execute(query)
  File "/code/callmeback/detail/log_helpers/postgre_conn.py", line 53, in execute
    return await super(LoggingConn, self).execute(query, *args, timeout=timeout)
  File "/usr/local/lib/python3.7/dist-packages/asyncpg/connection.py", line 273, in execute
    return await self._protocol.query(query, timeout)

Default command timeout is set for connection:

>>> conn = frame.f_locals['self']
>>> conn._config
ConnectionConfiguration(command_timeout=10.0, statement_cache_size=100, max_cached_statement_lifetime=300, max_cacheable_statement_size=15360)
>>> conn._protocol._get_timeout(None)
10.0

So I expect self._protocol.query() to throw TimeoutError of some kind, but related coroutine hanged for three days with no progress.

I will happily provide you with any additional information, if I can. My app has aiomonitor onboard, so if any info reachable that way will help you - please let me know.

I tried to fiddle with conn._protocol, but almost nothing of its state is reachable from aioconsole since it's Cython.

My suspection is that exception that leads to rollback is coming from in-transaction query that somehow leads connection in such a state that rollback() cannot be processed normally. I will provide you with information about exception once that problem triggers next time.

  • asyncpg version: 0.18.3
  • PostgreSQL version: 10.7
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    :
  • Python version: 3.7
  • Platform: Ubuntu 16.04
  • Do you use pgbouncer?: Yes (session-level pooling)
  • Did you install asyncpg with pip?: Yes
  • If you built asyncpg locally, which version of Cython did you use?: --
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : Yes, I am using uvloop

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with asyncpg/transaction.py at aexit and rollback, then follow connection.py execute() into _protocol.query(). Reproduce the hang with PostgreSQL 10.7, session-level pgbouncer pooling, and uvloop if possible; done means the rollback path either completes or surfaces the configured timeout instead of hanging.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.