tortoise / tortoise/tortoise-orm

Postgres connection string (dsn) with multi-host does not work

Open
#1,924 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
5.6k
Forks
516
Avg merge
2d 21h
Merged PRs (30d)
9

Description

Describe the bug

Postgres connection string (dsn) with multi-host does not work.

Example dsn:

postgres://fake-user:fake-password@127.1.2.3,127.1.2.4:12345/fake-db?target_session_attrs=read-write

To Reproduce

Run something like this:

import asyncio

from tortoise import Tortoise, transactions


async def main():
    config = {
        "connections": {
            # !!!
            # Using a DB_URL string with 2 hosts (primary + replica)
            "default": "postgres://postgres:qwerty123@host1.com,host2.com:5432/test",
        },
        "apps": {
            "my_app": {
                "models": ["__main__"],
                # If no default_connection specified, defaults to 'default'
                "default_connection": "default",
            }
        },
        "routers": ["path.router1", "path.router2"],
        "use_tz": False,
        "timezone": "UTC",
    }
    await Tortoise.init(config=config)

    async with transactions.in_transaction() as connection:
        await connection.execute_query("SELECT 1;")


if __name__ == '__main__':
    asyncio.run(main())
Traceback (most recent call last):
  File "contrib/python/starlette/starlette/routing.py", line 732, in lifespan
    async with self.lifespan_context(app) as maybe_state:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/tools/python3/Lib/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/fastapi/fastapi/routing.py", line 131, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ^^^^^^^^^^^^^^^^^^^^^
  File "contrib/tools/python3/Lib/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "src/api/app.py", line 30, in lifespan_v2
    await wait_until_connected()
  File "src/core/database.py", line 73, in wait_until_connected
    async with transactions.in_transaction() as connection:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/tortoise-orm/tortoise/backends/base/client.py", line 273, in __aenter__
    await self.ensure_connection()
  File "contrib/python/tortoise-orm/tortoise/backends/base/client.py", line 270, in ensure_connection
    await self.connection._parent.create_connection(with_db=True)
  File "contrib/python/tortoise-orm/tortoise/backends/asyncpg/client.py", line 59, in create_connection
    self._pool = await self.create_pool(password=self.password, **self._template)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/tortoise-orm/tortoise/backends/asyncpg/client.py", line 70, in create_pool
    return await asyncpg.create_pool(None, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/asyncpg/asyncpg/pool.py", line 418, in _async__init__
    await self._initialize()
  File "contrib/python/asyncpg/asyncpg/pool.py", line 445, in _initialize
    await first_ch.connect()
  File "contrib/python/asyncpg/asyncpg/pool.py", line 132, in connect
    self._con = await self._pool._get_new_connection()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/asyncpg/asyncpg/pool.py", line 517, in _get_new_connection
    con = await self._connect(
          ^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/asyncpg/asyncpg/connection.py", line 2421, in connect
    return await connect_utils._connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/asyncpg/asyncpg/connect_utils.py", line 1075, in _connect
    raise last_error or exceptions.TargetServerAttributeNotMatched(
  File "contrib/python/asyncpg/asyncpg/connect_utils.py", line 1049, in _connect
    conn = await _connect_addr(
           ^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/asyncpg/asyncpg/connect_utils.py", line 886, in _connect_addr
    return await __connect_addr(params, True, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/python/asyncpg/asyncpg/connect_utils.py", line 931, in __connect_addr
    tr, pr = await connector
             ^^^^^^^^^^^^^^^
  File "contrib/python/asyncpg/asyncpg/connect_utils.py", line 802, in _create_ssl_connection
    tr, pr = await loop.create_connection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/tools/python3/Lib/asyncio/base_events.py", line 1083, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/tools/python3/Lib/asyncio/base_events.py", line 1466, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/tools/python3/Lib/asyncio/base_events.py", line 905, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/tools/python3/Lib/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "contrib/tools/python3/Lib/socket.py", line 978, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

Expected behavior

It should work, like it works in asyncpg if we pass dsn url directly there.

Additional context

The bug is actually here during parsing:
https://github.com/tortoise/tortoise-orm/blob/develop/tortoise/backends/base/config_generator.py#L164

How it should be for Postgres and asyncpg:
https://github.com/MagicStack/asyncpg/blob/master/asyncpg/connect_utils.py#L272

I'd suggest to use this util inside if asyncpg engine is selected in the config.

Stack:

  • tortoise-orm == 0.23.0
  • asyncpg == 0.30.0
  • Postgres

Possible fix

I've made a small workaround in our project just for this case:

result = tortoise.backends.base.config_generator.expand_db_url(db_url=db_url)
result["credentials"]["host"] = result["credentials"]["host"].split(",")

config["connections"]["default"] = result

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at tortoise/backends/base/config_generator.py around line 164 and compare its PostgreSQL URL parsing with asyncpg/connect_utils.py around line 272. Reproduce the multi-host DSN case from the report and verify that the resulting configuration preserves both hosts so the connection behaves like asyncpg when given the DSN directly.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.