OSError: Too many open files from Manager process

OSError (Too many open files) is raised often in development environment.

Here is a part of full traceback

```
2025-12-16 17:33:53.219 ERROR ai.backend.manager.server [87933] Error inside event loop: Task exception was never retrieved
Traceback (most recent call last):
File "/Users/sh/lablup/bai-dev/src/ai/backend/common/plugin/monitor.py", line 87, in capture_exception
File "/Users/sh/lablup/bai-dev/src/ai/backend/manager/plugin/error_monitor.py", line 73, in capture_exception
File "/Users/sh/.pyenv/versions/3.13.7/lib/python3.13/contextlib.py", line 214, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/src/ai/backend/manager/models/utils.py", line 144, in begin
async with self.connect() as _bind:
~~~~~~~~~~~~^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/ext/asyncio/base.py", line 66, in __aenter__
return await self.start(is_ctxmanager=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/ext/asyncio/engine.py", line 157, in start
await (greenlet_spawn(self.sync_engine.connect))
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 127, in greenlet_spawn
result = context.throw(*sys.exc_info())
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/future/engine.py", line 412, in connect
return super(Engine, self).connect()
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 3325, in connect
return self._connection_cls(self, close_with_result=close_with_result)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
else engine.raw_connection()
~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 3404, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 3371, in _wrap_pool_connect
return fn()
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 327, in connect
return _ConnectionFairy._checkout(self)
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 894, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 493, in checkout
rec = pool._do_get()
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/impl.py", line 145, in _do_get
with util.safe_reraise():
~~~~~~~~~~~~~~~~~^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
~~~~~~~~~~~~~^
exc_value,
^^^^^^^^^^
with_traceback=exc_tb,
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
return self._create_connection()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 273, in _create_connection
return _ConnectionRecord(self)
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 388, in __init__
self.__connect()
~~~~~~~~~~~~~~^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 690, in __connect
with util.safe_reraise():
~~~~~~~~~~~~~~~~~^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
~~~~~~~~~~~~~^
exc_value,
^^^^^^^^^^
with_traceback=exc_tb,
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/pool/base.py", line 686, in __connect
self.dbapi_connection = connection = pool._invoke_creator(self)
~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/engine/create.py", line 574, in connect
return dialect.connect(*cargs, **cparams)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 598, in connect
return self.dbapi.connect(*cargs, **cparams)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 800, in connect
await_only(self.asyncpg.connect(*arg, **kw)),
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 70, in await_only
return current.parent.switch(awaitable)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 122, in greenlet_spawn
value = await result
^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/asyncpg/connection.py", line 2443, in connect
return await connect_utils._connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<22 lines>...
)
^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/asyncpg/connect_utils.py", line 1209, in _connect
addrs, params, config = _parse_connect_arguments(**kwargs)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/asyncpg/connect_utils.py", line 893, in _parse_connect_arguments
addrs, params = _parse_connect_dsn_and_args(
~~~~~~~~~~~~~~~~~~~~~~~~~~~^
dsn=dsn, host=host, port=port, user=user,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<4 lines>...
krbsrvname=krbsrvname, gsslib=gsslib,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
service=service, servicefile=servicefile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/sh/lablup/bai-dev/dist/export/python/virtualenvs/python-default/3.13.7/lib/python3.13/site-packages/asyncpg/connect_utils.py", line 781, in _parse_connect_dsn_and_args
ssl.load_cert_chain(
~~~~~~~~~~~~~~~~~~~^
sslcert,
^^^^^^^^
keyfile=sslkey,
^^^^^^^^^^^^^^^
password=lambda: sslpassword
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
OSError: [Errno 24] Too many open files
2025-12-16 17:33:53.225 WARNING ai.backend.manager.api.stream [87933] stream_conn_tracker_gc(): error while connecting to Etcd server, retrying...
2025-12-16 17:33:53.334 WARNING opentelemetry.exporter.otlp.proto.grpc.exporter [87933] Transient error StatusCode.UNAVAILABLE encountered while exporting logs to 127.0.0.1:4317, retrying in 1.06s.
2025-12-16 17:33:59.397 ERROR opentelemetry.exporter.otlp.proto.grpc.exporter [87933] Failed to export logs to 127.0.0.1:4317, error code: StatusCode.UNAVAILABLE

thread '' panicked at /Users/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/reqwest-0.12.15/src/blocking/client.rs:1048:38:
Client::new(): reqwest::Error { kind: Builder, source: Os { code: 24, kind: Uncategorized, message: "Too many open files" } }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2025-12-16 17:34:04.401 WARNING opentelemetry.exporter.otlp.proto.grpc.exporter [87933] Transient error StatusCode.UNAVAILABLE encountered while exporting logs to 127.0.0.1:4317, retrying in 0.90s.
2025-12-16 17:34:12.074 ERROR opentelemetry.exporter.otlp.proto.grpc.exporter [87933] Failed to export logs to 127.0.0.1:4317, error code: StatusCode.UNAVAILABLE
2025-12-16 17:34:17.079 WARNING opentelemetry.exporter.otlp.proto.grpc.exporter [87933] Transient error StatusCode.UNAVAILABLE encountered while exporting logs to 127.0.0.1:4317, retrying in 0.94s.
2025-12-16 17:34:24.372 ERROR opentelemetry.exporter.otlp.proto.grpc.exporter [87933] Failed to export logs to 127.0.0.1:4317, error code: StatusCode.UNAVAILABLE
2025-12-16 17:34:29.375 WARNING opentelemetry.exporter.otlp.proto.grpc.exporter [87933] Transient error StatusCode.UNAVAILABLE encountered while exporting logs to 127.0.0.1:4317, retrying in 0.88s.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1765874070.914053 21664166 socket_utils_common_posix.cc:482] socket(30, 1, 0) returned -1 with error: |Too many open files|. This process might not have a sufficient file descriptor limit for the number of connections grpc wants to open (which is generally a function of the number of grpc channels, the lb policy of each channel, and the number of backends each channel is load balancing across).
```

JIRA Issue: BA-3389