docling-project / docling-project/docling
[BUG] docling-parse crashes inside its own worker threads: invalid free (SIGABRT) and invalid read (SIGSEGV), 8 occurrences in one day — macOS arm64, Python 3.12
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Bug
`docling-parse`'s native extension (`pdf_parsers.cpython-312-darwin.so`) crashes the whole process on some PDFs, in **two different ways that share one signature: multiple `docling-parse` worker threads running concurrently inside the extension.**
In one day on a single machine I collected **eight** native crashes. Every one of them has 4–5 threads inside `pdf_parsers` at the moment of death:
| Time | Signal | Fault | Threads | In `pdf_parsers` |
|---|---|---|---|---|
| 18:39 | SIGABRT | invalid free (`free_tiny_botch`) | 40 | 5 |
| 18:41 | SIGSEGV | invalid read at `0x0` | 29 | 5 |
| 18:42 | SIGABRT | invalid free | 28 | 5 |
| 18:43 | SIGSEGV | invalid read at `0x0` | 27 | 4 |
| 23:27 | SIGABRT | invalid free | 23 | 5 |
| 23:28 | SIGABRT | invalid free | 23 | 5 |
| 23:34 | SIGABRT | invalid free | 23 | 5 |
| 00:48 | SIGSEGV | invalid read at `0x100000000` | 29 | 5 |
Two symptoms, one cause. **Invalid free** is corrupted heap metadata; **invalid read of a null or wild pointer** is a corrupted or torn pointer. Both are what a data race on shared state produces, and which one you get depends on timing.
**SIGABRT shape** — macOS's allocator catching a free of something that is not a valid live allocation:
```
Exception Type: EXC_CRASH (SIGABRT) — "Abort trap: 6"
asi: libsystem_c.dylib "abort() called"
2 libsystem_c.dylib abort + 124
3 libsystem_malloc.dylib malloc_vreport + 892
4 libsystem_malloc.dylib malloc_zone_error + 100
5 libsystem_malloc.dylib free_tiny_botch + 40
6 docling_parse pdf_parsers.cpython-312-darwin.so
```
**SIGSEGV shape** — and here is the part I think matters most. Two worker threads are deep inside the extension at once, and **their bottom six frames are identical** while their top three differ:
```
Thread 25 CRASHED Thread 26 (running concurrently)
pdf_parsers +4126020 <-- fault pdf_parsers +4230872
pdf_parsers +4125172 pdf_parsers +4230164
pdf_parsers +4266324 pdf_parsers +4264816
pdf_parsers +4378516 ─┐ pdf_parsers +4378516 ─┐
pdf_parsers +4474492 │ pdf_parsers +4474492 │
pdf_parsers +5652564 │ identical pdf_parsers +5652564 │ identical
pdf_parsers +5595844 │ tail pdf_parsers +5595844 │ tail
pdf_parsers +5882920 │ pdf_parsers +5882920 │
pdf_parsers +5887964 ─┘ pdf_parsers +5887964 ─┘
_pthread_start _pthread_start
EXC_BAD_ACCESS (SIGSEGV), KERN_INVALID_ADDRESS at 0x0000000100000000
esr: 0x92000006 (Data Abort) byte read Translation fault
```
That shared tail is a thread-pool worker entry point: same dispatch, two different work items, running at the same time. A third thread sits in `std::condition_variable::wait` on the same tail — the idle pool worker.
Two further threads in that same crash are in **C++ exception teardown at the same time**, from the same pool entry:
```
Thread 27: __cxa_decrement_exception_refcount → std::logic_error::~logic_error
→ compute_gcc_empty_string_storage → dlopen (!)
Thread 28: __cxa_decrement_exception_refcount → std::logic_error::~logic_error
→ __cxa_guard_acquire (blocked on the same static-init guard)
```
Two pool threads destroying `std::logic_error` exceptions concurrently, racing on a libc++abi static initializer — one of them ending up in `dlopen` during unwinding. Whatever the pool does on a malformed PDF, it appears to throw from several threads at once.
### Steps to reproduce
```python
from docling.document_converter import DocumentConverter
DocumentConverter().convert(".pdf") # → SIGABRT (134) or SIGSEGV (139)
```
with `PdfPipelineOptions(do_ocr=False)` and `settings.inference.compile_torch_models = False`.
I regret that I **cannot share the triggering documents** — they are confidential customer files, and I have not yet found a public PDF that reproduces it. I'm filing because the thread evidence above localizes this without the input: the crash is inside `docling-parse`'s own concurrency, and it reproduces roughly hourly under normal use.
**Importantly, the caller is single-threaded.** This was reproduced standalone — no web server, no job queue, no concurrent conversions, one process converting one file, a single-threaded caller — and it still crashes with five threads inside `pdf_parsers`. The concurrency is entirely internal to `docling-parse`.
What I ruled out:
| Hypothesis | Evidence against |
|---|---|
| OOM / memory exhaustion | Peak RSS a few hundred MB; an OOM kill is SIGKILL/137 and produces no crash report. `free_tiny_botch` is a validity check, not an allocation failure |
| Concurrency **in calling code** | Reproduced single-process, single-file, single-threaded caller. (Concurrency *inside* `docling-parse` is the hypothesis, not the thing ruled out) |
| `torch.compile` codegen at parse time | `settings.inference.compile_torch_models = False` set before pipeline construction |
| OCR engines (EasyOCR / RapidOCR / ocrmac) | `do_ocr=False` on the failing path |
### Questions
1. Is `docling-parse`'s internal page/parse worker pool expected to be thread-safe, and is there shared mutable state across those workers?
2. Is there a way to force it to single-threaded from the Python side? I could confirm the diagnosis immediately by setting the pool to one thread and seeing whether both crash shapes disappear.
**I'm happy to dig further.** I can run the failing files under `MallocStackLogging`, guard malloc (`libgmalloc`), or a TSan/ASan build of `docling-parse` and post the race or the offending allocation backtrace. Tell me which is most useful and I'll run it.
### Docling version
```
Docling version: 2.123.0
Docling Core version: 2.91.0
Docling IBM Models version: 3.13.3
Docling Parse version: 7.16.0
Python: cpython-312 (3.12.1)
Platform: macOS-26.3-arm64-arm-64bit
```
Caveat on version binding: macOS redacts virtualenv paths in crash reports (`/Users/USER/*/pdf_parsers...`), so I cannot bind each individual report to a specific environment. The above is the pinned and installed version in the environment where I reproduced this standalone; the same machine also has 7.14.0 and 6.2.0 installed in unrelated environments.
### Python version
```
Python 3.12.1
```
(python.org framework build, arm64)
---
Possibly related, but I don't believe any is the same bug: #3671 and #3345 (`std::bad_alloc` — memory *exhaustion* on large documents, whereas this is corruption at low RSS), #2684 (native crash on macOS arm64, but Python 3.14), #2109 (hangs rather than crashes).
`PyPdfiumDocumentBackend` avoids this code path entirely and is my current candidate workaround.
Contributor guide
Research direction
Start with the standalone DocumentConverter().convert(".pdf") path using PdfPipelineOptions(do_ocr=False) and compile_torch_models disabled, then inspect the docling-parse native extension's internal worker-pool entry point. Use the reported macOS arm64 crash traces and, if possible, TSan/ASan or guard malloc; done means eliminating the concurrent invalid-free and invalid-read crashes or documenting a reliable single-threaded control.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100