MagicStack / MagicStack/asyncpg

`TypeError: cannot unpack non-iterable NoneType object` when using docker bridge network

Abierto
#1,030 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Lenguaje dominante
Python
Estrellas
8.1k
Forks
468
Sin PR fusionados en 30 d

Descripción

  • asyncpg version: 0.27.0
  • PostgreSQL version: 12
  • Python version: 3.8
  • Platform: Ubuntu 22
  • Do you use pgbouncer?: no
  • Did you install asyncpg with pip?: yes

It seems that asyncpg is not working correctly when using an host defined using a bridge network in docker. I am actually not sure the error comes from asyncpg or docker, any idea here could be really helpful.
Opening any new connection using an asyncpg connection string will result in the following error TypeError: cannot unpack non-iterable NoneType object, but only when using a network alias from a docker bridge network.

More Details + how to reproduce

I have a docker-compose.yml file looking like: (simplified for the sake of this issue)

version: '3.4'

services:
  postgresdb:
    image: postgres:12
    networks:
      proxy_network:
        aliases:
          - postgresdb_host
    env_file:
      - db.env
    ports:
      - 3333:5432
    volumes:
      - ./db/postgres:/docker-entrypoint-initdb.d
      - ./storage/postgres:/var/lib/postgresql/data

  app_one_api:
    build:
      context: ${APP_ONE_DIR}
      target: base
    ports:
      - 5000:5000
    networks:
      proxy_network:
        aliases:
          - app_one_host
    restart: on-failure
    command: bash -c "flask run --port 5000 --host 0.0.0.0"
    volumes:
      - ${APP_ONE_DIR}:/app
      - ./storage/app_one/:/storage/
    depends_on:
      - postgresdb

  app_two_api:
    build:
      context: ${APP_TWO_DIR}
      target: prod
    ports:
      - 1234:8383
    networks:
      proxy_network:
        aliases:
          - app_two_host
    restart: on-failure
    entrypoint: bash -c "python main.py"
    volumes:
      - ${APP_TWO_DIR}:/app
      - ./storage/app_two/:/storage/
    depends_on:
      - postgresdb

networks:
  proxy_network:
    driver: bridge

Where app_one is a Flask application using SQLAlchemy and psycopg2 (synchronous session, no asyncpg there), and app_two is a fastAPI application using SQLAlchemy and asyncpg.

  • When I use postgresql://postgres:postgres@postgresdb_host:5432/my_db_name as an URI for the Flask application, everything is working well, meaning my proxy_network is configured correctly between app_one and postgresdb (and db is properly running)
  • When I run my app_two outside of docker, but still run postgresdb container, I can properly use postgresql+asyncpg://postgres:postgres@127.0.0.1:3333/my_db_name and it works. (external host / ports), meaning the database container is working correctly
  • When I do a HTTP call from my app_two container, to an endpoint of app_one, using http://app_one_host:5000 as URI, it works correctly, meaning app_two and app_one are also properly linked to the bridge network.

However, using postgresql+asyncpg://postgres:postgres@postgresdb_host:5432/my_db_name as my database URI from within the docker container of app_two, here is the error I get when trying to open any new connection:

   File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 825, in __connect_addr
     tr, pr = await compat.wait_for(connector, timeout=timeout)
 TypeError: cannot unpack non-iterable NoneType object
see full error stacktrace: 
File "/app/app/db/session.py", line 85, in check_dbs
     async with self.engines[bind].begin() as connection:
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/base.py", line 66, in __aenter__
     return await self.start(is_ctxmanager=True)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/engine.py", line 599, in start
     await self.conn.start(is_ctxmanager=is_ctxmanager)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/engine.py", line 157, in start
     await (greenlet_spawn(self.sync_engine.connect))
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 126, in greenlet_spawn
     result = context.throw(*sys.exc_info())
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/future/engine.py", line 406, in connect
     return super(Engine, self).connect()
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3320, in connect
     return self._connection_cls(self, close_with_result=close_with_result)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
     else engine.raw_connection()
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3399, in raw_connection
     return self._wrap_pool_connect(self.pool.connect, _connection)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3366, in _wrap_pool_connect
     return fn()
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 327, in connect
     return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
     fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 493, in checkout
     rec = pool._do_get()
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
     self._dec_overflow()
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
     compat.raise_(
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
     raise exception
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
     return self._create_connection()
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 273, in _create_connection
     return _ConnectionRecord(self)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 388, in __init__
     self.__connect()
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 691, in __connect
     pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
     compat.raise_(
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
     raise exception
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
     self.dbapi_connection = connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 574, in connect
     return dialect.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 598, in connect
     return self.dbapi.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 780, in connect
     await_only(self.asyncpg.connect(*arg, **kw)),
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
     return current.driver.switch(awaitable)
   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
     value = await result
   File "/usr/local/lib/python3.8/site-packages/asyncpg/connection.py", line 2092, in connect
     return await connect_utils._connect(
   File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 881, in _connect
     return await _connect_addr(
   File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 773, in _connect_addr
     return await __connect_addr(params, timeout, True, *args)
   File "/usr/local/lib/python3.8/site-packages/asyncpg/connect_utils.py", line 825, in __connect_addr
     tr, pr = await compat.wait_for(connector, timeout=timeout)
 TypeError: cannot unpack non-iterable NoneType object

With all those experiments I did, I cannot find what is the issue, except that it happens only when using asyncpg + bridge network. Bridge network with psycopg2 works fine, as well as asyncpg without bridge network hosts. Combining both fails.

I tried to add as much info as possible on my issue and how to reproduce, but feel free to ask anything I may have forgotten!

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Línea de trabajo

Empieza reproduciendo la conexión desde app_two mediante la red bridge de docker-compose.yml y la URI postgresdb_host. Lee asyncpg/connect_utils.py alrededor de __connect_addr y compáralo con la conexión funcional a 127.0.0.1; el punto de llamada de la aplicación está en app/db/session.py. Se considera terminado cuando la ruta de conexión indicada ya no provoca el TypeError de desempaquetado y se han verificado la causa y el comportamiento.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
docker, docker-compose, postgresql, python
Área
backend, databases, networking
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
30/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.