A stalled connection to the embedding service blocks the whole indexing batch for request_timeout

Describe the Bug

NetworkEmbeddings._get_embedding() passes a single value as the timeout of the embedding request:

response = niquests.post(
    f'{emconf.base_url.removesuffix("/")}/embeddings',
    json=data,
    timeout=emconf.request_timeout,
    ...
)

With one value it applies to the read, and there is no separate bound on establishing the connection. When a connection to the embedding service stalls during the TLS/HTTP2 handshake — TCP connects, the handshake never completes — the worker sits in recv for the whole request_timeout, which defaults to 1800 s.

That is expensive because of how the work is batched: files_indexing_thread dispatches the batch across workers and then waits for every one of them (Waiting for file chunk 1/8 future to complete). One stalled handshake holds up the entire batch, and no further queue items are fetched until it returns.

Evidence

Captured with py-spy on a stalled worker:

Thread 1629 (idle): "MainThread"
    sync_recv_gro (urllib3/contrib/ssa/_gro.py:205)
    __exchange_until (urllib3/backend/hface.py:1018)
    _post_conn (urllib3/backend/hface.py:712)
    connect (urllib3/connection.py:894)
    _validate_conn (urllib3/connectionpool.py:820)
    _make_request (urllib3/connectionpool.py:1281)
    urlopen (urllib3/connectionpool.py:1821)
    send (niquests/adapters.py:927)
    request (niquests/api.py:122)
    post (niquests/api.py:401)
    _get_embedding (context_chat_backend/network_em.py:99)
    embed_documents (context_chat_backend/network_em.py:170)
    add_texts (langchain_postgres/vectorstores.py:885)
    add_indocuments (context_chat_backend/vectordb/pgvector.py:186)
    _process_sources (context_chat_backend/chain/ingest/injest.py:420)
    embed_sources (context_chat_backend/chain/ingest/injest.py:446)

It is inside connect(), in the handshake exchange, not in reading a response.

The signature in the logs is a batch whose workers all finish together at the timeout rather than at their own pace:

Dispatching 8 file chunk(s) and 0 provider chunk(s)
Waiting for file chunk 1/8 future to complete
Subprocess ... finished in 302262.22 ms
Subprocess ... finished in 302492.21 ms
Subprocess ... finished in 303569.71 ms
Subprocess ... finished in 305397.98 ms

Worth noting how well the service was otherwise: a direct request to the same endpoint from the same container returned in 0.28 s for one text and 4.37 s for 100 texts of 2000 characters, and 24 concurrent fresh connections all completed in 0.6 s. The stall is intermittent and only shows up under the sustained load of the real indexing run.

Impact

Measured on our deployment, before and after adding a connection timeout, everything else unchanged:

Roughly a twelvefold difference. Before the change the run looked stopped; operators reasonably read it as a hang rather than as slow progress.

Suggested fix

Give the connection its own bound, so a stalled handshake fails quickly and the retry that _get_embedding() already implements can land on a healthy connection:

timeout=(HTTP_CONNECT_TIMEOUT, emconf.request_timeout),

PR follows.

Setup Details

Nextcloud version: 34.0.3
context_chat / context_chat_backend: 5.4.0 / 5.4.1
Deployment: manual-install deploy daemon, four backend instances, external embedding service, PostgreSQL 18 with pgvector

Start in context_chat_backend/network_em.py at NetworkEmbeddings._get_embedding() and inspect how its request timeout and retry behavior work. Trace the callers in context_chat_backend/chain/ingest/injest.py and context_chat_backend/vectordb/pgvector.py to understand the batch impact. Done means a stalled connection has its own bound while the existing response timeout and retry behavior remain intact.