eikek / eikek/docspell

joex: concurrent unoconv against shared LibreOffice listener deadlocks the processing pool

Open
#3,345 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Elm
Stars
2.3k
Forks
184
Avg merge
8h 2m
Merged PRs (30d)
4

Description

## Problem

When multiple office documents (e.g. `.xlsx`) are processed in parallel, joex can deadlock the entire conversion pool for hours. Workers sit on `Waiting for command to terminate…` while `unoconv` never finishes.

This is separate from wanting a store-only upload API (#3344): even when conversion *is* desired, concurrent office converts are unsafe with the current setup.

## Root cause

Docspell’s Docker/joex setup runs a **single** LibreOffice listener via `unoconv -l`. Each office file is converted with a client `unoconv -f pdf …` that talks to that listener.

With `docspell.joex.scheduler.pool-size > 1` (e.g. `5`), several `process-item` jobs call `unoconv` **at the same time** against that one `soffice.bin`. LibreOffice/UNO headless is not safe for concurrent document conversion. In practice:

- multiple `unoconv` client processes stay alive for hours
- `soffice` holds several input files open at once
- URP reader/writer threads sit idle (`futex_wait`)
- no `out.pdf` is produced
- all joex workers in the pool block on conversion → queue stalls

Observed in production with **tiny** Excel files (~8–16 KB), so this is not “huge file” load — it is **concurrency**.

The configured `docspell.joex.convert.unoconv.command.timeout` (default `2 minutes`) and UI **Cancel** do not reliably free the pool once UNO is wedged (child `unoconv`/LibreOffice sessions can remain after cancel).

## How to confirm

Inside the joex container while stuck:

```bash
ps -eo pid,etime,cmd | grep -E 'unoconv|soffice'
ls -l /proc/$(pgrep -n soffice.bin)/fd | grep unoconv
# several infiles open at once → concurrent LO deadlock
```

A manual `timeout 25 unoconv …` against one of the open infiles also hangs while the listener is in this state.

## Related

- #3293 — earlier unoconv/soffice connectivity issue; @eikek noted we rely on `unoconv -l` and should move to [unoserver](https://github.com/unoconv/unoserver/) at some point
- #3344 — store-only upload (`process: false`) avoids convert for *reference* files, but does not fix concurrent conversion for files that still need PDF

## Possible directions

1. **Near-term:** serialize office conversion in Docspell (mutex/semaphore around `unoconv`), independent of scheduler pool size; harden `SysExec` timeout to destroy the process before joining log fibers / use forcible destroy
2. **Ops workaround:** set `scheduler.pool-size=1` (stops the hang; reduces throughput for all tasks)
3. **Longer-term (preferred upstream direction from #3293):** migrate `unoconv` → `unoserver` / `unoconverter` (server queues converts sequentially per instance; scale with multiple servers on distinct ports if needed)

Happy to help with (1) and/or (3).

Contributor guide

Open the contributing guide

Research direction

Start by tracing joex's process-item office-conversion path and the SysExec timeout handling described in the issue; no source file or test is named. Reproduce with scheduler.pool-size greater than 1 and concurrent office files, then verify that conversions complete without wedging the pool and that cancellation or timeout releases blocked work.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.