joex: concurrent unoconv against shared LibreOffice listener deadlocks the processing pool
- Dominant language
- Elm
- Stars
- 2.3k
- Forks
- 184
- Avg merge
- 8h 2m
- Merged PRs (30d)
- 4
Description
## Problem
When multiple office documents (e.g. `.xlsx`) are processed in parallel, joex can deadlock the entire conversion pool for hours. Workers sit on `Waiting for command to terminate…` while `unoconv` never finishes.
This is separate from wanting a store-only upload API (#3344): even when conversion *is* desired, concurrent office converts are unsafe with the current setup.
## Root cause
Docspell’s Docker/joex setup runs a **single** LibreOffice listener via `unoconv -l`. Each office file is converted with a client `unoconv -f pdf …` that talks to that listener.
With `docspell.joex.scheduler.pool-size > 1` (e.g. `5`), several `process-item` jobs call `unoconv` **at the same time** against that one `soffice.bin`. LibreOffice/UNO headless is not safe for concurrent document conversion. In practice:
- multiple `unoconv` client processes stay alive for hours
- `soffice` holds several input files open at once
- URP reader/writer threads sit idle (`futex_wait`)
- no `out.pdf` is produced
- all joex workers in the pool block on conversion → queue stalls
Observed in production with **tiny** Excel files (~8–16 KB), so this is not “huge file” load — it is **concurrency**.
The configured `docspell.joex.convert.unoconv.command.timeout` (default `2 minutes`) and UI **Cancel** do not reliably free the pool once UNO is wedged (child `unoconv`/LibreOffice sessions can remain after cancel).
## How to confirm
Inside the joex container while stuck:
```bash
ps -eo pid,etime,cmd | grep -E 'unoconv|soffice'
ls -l /proc/$(pgrep -n soffice.bin)/fd | grep unoconv
# several infiles open at once → concurrent LO deadlock
```
A manual `timeout 25 unoconv …` against one of the open infiles also hangs while the listener is in this state.
## Related
- #3293 — earlier unoconv/soffice connectivity issue; @eikek noted we rely on `unoconv -l` and should move to [unoserver](https://github.com/unoconv/unoserver/) at some point
- #3344 — store-only upload (`process: false`) avoids convert for *reference* files, but does not fix concurrent conversion for files that still need PDF
## Possible directions
1. **Near-term:** serialize office conversion in Docspell (mutex/semaphore around `unoconv`), independent of scheduler pool size; harden `SysExec` timeout to destroy the process before joining log fibers / use forcible destroy
2. **Ops workaround:** set `scheduler.pool-size=1` (stops the hang; reduces throughput for all tasks)
3. **Longer-term (preferred upstream direction from #3293):** migrate `unoconv` → `unoserver` / `unoconverter` (server queues converts sequentially per instance; scale with multiple servers on distinct ports if needed)
Happy to help with (1) and/or (3).
Contributor guide
Research direction
Start by tracing joex's process-item office-conversion path and the SysExec timeout handling described in the issue; no source file or test is named. Reproduce with scheduler.pool-size greater than 1 and concurrent office files, then verify that conversions complete without wedging the pool and that cancellation or timeout releases blocked work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100