A batch waits for its slowest worker, so one slow source stalls indexing and blocks the next fetch

Describe the Bug

A batch makes no progress until its slowest worker returns, so one slow source stalls the whole batch and no new queue items are fetched meanwhile.

files_indexing_thread submits the chunks and then walks the futures in submission order:

file_futures = [executor.submit(_load_sources, chunk) for chunk in file_chunks]
...
for i, future in enumerate(file_futures):
    LOGGER.debug('Waiting for file chunk %d/%d future to complete', i + 1, len(file_futures))
    files_result.update(future.result())

Results are reported and the queue rows deleted only after that loop, so nothing a fast worker produced is committed until the slowest one is done.

That is fine when workers finish in comparable times. It is expensive when they do not, and they often do not: in our runs the workers of a single batch finished anywhere between 1.4 s and 300 s. The 300 s ones were an embedding request that never got a response, retried up to three times against request_timeout.

How visible this is

Doubling doc_indexing_batch_size from 32 to 64 halved our throughput, from ~37 to ~15–20 documents/min. Each worker then carries twice as many sources, so the chance that a batch contains a slow one goes up and the tail it imposes doubles. We reverted it.

The default request_timeout of 1800 s compounds it: before we lowered it, a single unresponsive embedding request held a batch for ~35 minutes (3 attempts plus back-offs), during which all four of our backend instances sat idle. Lowering it to 30 s — healthy requests answer in about 6 s — was the single largest improvement we made, from ~5 to ~60–100 documents/min.

Suggested direction

Two things would help independently:

• Consume the futures as they complete (concurrent.futures.as_completed) and report results incrementally, so a slow worker delays only its own sources rather than the batch and the next fetch.
• Reconsider the default request_timeout. 1800 s is a long time to hold a worker for a live indexing pipeline; something in the tens of seconds, with the retry budget in mind, matches how the embedding services actually behave.

I have not sent a patch for the first one because changing the batch bookkeeping touches the result and deletion paths, and I would rather not guess at the invariants you want there. Happy to write it if you tell me which shape you prefer.

Setup Details

Nextcloud version: 34.0.3
context_chat / context_chat_backend: 5.4.0 / 5.4.1
Deployment: manual-install deploy daemon, four backend instances, external embedding service, PostgreSQL 18 with pgvector, ~1.5 M eligible files

Start at files_indexing_thread and trace how _load_sources futures feed the result and queue-row deletion paths. Compare the current submission-order loop with completion-order handling, and review how request_timeout and retries affect the batch. Done means fast results are reported and committed without waiting for the slowest worker, while bookkeeping remains correct.