duckdb / duckdb/duckdb-python

Access violation (0xC0000005) in _duckdb.cp313-win_amd64.pyd during concurrent read-only queries; same defect on 1.4.4 and 1.5.5, two WER buckets; asking for symbolication of three offsets

Open
#613 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
187
Forks
112
Avg merge
13h 29m
Merged PRs (30d)
17

Description

### What happens?

A multi-process worker pool doing concurrent **read-only** queries against a DuckDB file with the `spatial` extension crashes repeatedly with `STATUS_ACCESS_VIOLATION` (`0xC0000005`) inside the DuckDB extension binary. There is no interrupt, no signal, no cancellation, and no writer: the worker processes simply die mid-query and the supervisor respawns them.

It reproduces on **two different machines, two DuckDB versions (1.4.4 and 1.5.5), two CPython patch levels and two unrelated workloads**. I have minidumps and `cdb` stacks for both versions, and I believe **the 1.4.4 and 1.5.5 crashes are one defect** even though Windows Error Reporting buckets them as two (see "One defect, two WER buckets" below).

I cannot supply a minimal reproducer and I cannot get past this without maintainer-side symbolication, because the published Windows wheel ships **no PDBs**. Concrete asks are at the bottom.

#### Frequency

On the 1.5.5 host, a single ~7 minute run of the workload produced **14** `Application Error` / Event ID 1000 records for `python.exe`, and **38** in one day. Each crash kills one worker within seconds of that worker starting; the parent survives, so from the outside "nothing is down".

#### Fault site (1.5.5)

Exception `c0000005`, `INVALID_POINTER_READ`, faulting module `_duckdb.cp313-win_amd64.pyd`.

```
ExceptionAddress: 00007ffdcbf5c020 (_duckdb_cp313_win_amd64+0x00000000018dc020)
ExceptionCode: c0000005 (Access violation)
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005__duckdb.cp313-win_amd64.pyd!Unknown

00007ffd`cbf5c020 8a01 mov al, byte ptr [rcx] ds:00028910`ebfa612f=??
00007ffd`cbf5c022 3a0411 cmp al, byte ptr [rcx+rdx]

rax=00028910ebfa612f rbx=000000000000007d rcx=00028910ebfa612f
rdx=ffffffffff86dd00 rsi=000000000000007d rdi=0000000000000001
rip=00007ffdcbf5c020 rsp=00000058e537d818 rbp=0000000000000000
r8=0000000000001f00
```

The `mov al,[rcx]` / `cmp al,[rcx+rdx]` pair is the shape of an **inlined byte-compare loop**. Note `rdx` holds a sign-extended-negative value where a pointer or offset argument belongs.

Caveat on the reported fault address: `cdb` prints `Attempt to read from address ffffffffffffffff`, but that is its rendering of an address the minidump did not record. The actual bad pointer is `rcx = 0x00028910ebfa612f`.

Fault offsets observed in one day on this host (Event-1000 `Properties[7]`):

| module offset | count |
|---|---|
| `0x00000000018dc020` | 24 |
| `0x00000000018dc060` | 11 |
| `0x00000000018dc0a0` | 1 |
| `0x000000000058b2fa` | 1 |

Three fault sites `0x40` apart inside one region is consistent with an unrolled byte-compare loop.

Stack, 1.5.5 (faulting thread, index 27 of 47). **Every `duckdb::` name below is the nearest preceding *export* plus a large offset — the wheel ships no PDBs, so the module attribution is definitive but the function names are almost certainly not the functions executing.** Treat repeated names as "same region of the binary".

```
_duckdb!duckdb::FunctionBinder::BindSortedAggregate+0x1d76a0 <-- fault
_duckdb!duckdb::MetadataBlock::operator=+0x54662
_duckdb!duckdb::MetadataBlock::operator=+0x21bec
_duckdb!duckdb::MetadataBlock::operator=+0x642f4
_duckdb!duckdb::VectorOperations::TryCast+0x488d
_duckdb!duckdb::ExpressionExecutor::ExecuteExpression+0x5b1
_duckdb!duckdb::FunctionBinder::BindSortedAggregate+0x40be5
_duckdb!duckdb::InterruptState::Callback+0x16d0
_duckdb!duckdb::InterruptState::Callback+0x2aa8
_duckdb!duckdb::StreamQueryResult::ExecuteTask+0x11a
_duckdb!duckdb::PendingQueryResult::ExecuteTask+0x59
_duckdb!duckdb::FunctionData::SupportStatementCache+0x874
_duckdb!duckdb::FileSystem::IsPipe+0xdb5
_duckdb!duckdb::FunctionData::SupportStatementCache+0x2539
_duckdb!duckdb::ArenaAllocator::GetTail+0x1d8f4
_duckdb!duckdb::ArenaAllocator::GetTail+0x26d6
_duckdb!duckdb::NumericValueUnion::GetReferenceUnsafe+0x8168
python313+0x2eb52
```

#### Fault site (1.4.4, a different machine)

```
ExceptionAddress: 00007ff813271340 (VCRUNTIME140!memcmp+0x0000000000000050)
ExceptionCode: c0000005 (Access violation)
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_VCRUNTIME140.dll!memcmp

00007ff8`13271340 488b01 mov rax, qword ptr [rcx] ds:000001dc`a53008b0=????????????????

rax=000001dca52ffe6c rbx=0000000000000000 rcx=000001dca53008b0
rdx=fffffffffffff5bc rsi=0000000000000000 rdi=0000000000000001
r8=0000000000000024 rbp=0000000000000000
```

```
VCRUNTIME140!memcmp+0x50 <-- fault
_duckdb!duckdb::MetadataBlock::operator=+0x572f1
_duckdb!duckdb::MetadataBlock::operator=+0x26451
_duckdb!duckdb::MetadataBlock::operator=+0x69d9d
_duckdb!duckdb::BufferedFileWriter::WriteData+0x2b1e
_duckdb!duckdb::ExpressionExecutor::ExecuteExpression+0x5be
_duckdb!duckdb::FunctionBinder::BindSortedAggregate+0x184d5
_duckdb!duckdb::InterruptState::Callback+0x16b0
_duckdb!duckdb::InterruptState::Callback+0x22cb
_duckdb!duckdb::VectorOperations::TryCast+0x1606
ucrtbase!thread_start+0x30
kernel32!BaseThreadInitThunk+0x17
ntdll!RtlUserThreadStart+0x2c
```

Second-sourced: an independently captured 1.4.4 minidump from 2026-08-31 gives the same `VCRUNTIME140!memcmp+0x50` and the same bucket.

#### One defect, two WER buckets

I would ask you not to dedup these apart on the bucket names. Evidence that they are the same defect:

- **Seven middle frames correspond, in the same order**, across a minor-version bump: `MetadataBlock::operator=` x3 -> a mid-level frame -> `ExpressionExecutor::ExecuteExpression` -> `FunctionBinder::BindSortedAggregate` -> `InterruptState::Callback` x2.
- `ExpressionExecutor::ExecuteExpression` **+0x5be (1.4.4) vs +0x5b1 (1.5.5)** — 13 bytes apart. *Hedged:* this only holds if the nearest-preceding export is the same real function in both builds, which is unverifiable without PDBs.
- **The faulting operation is a byte-wise comparison entered with a garbage pointer in both.** 1.4.4 calls out to `VCRUNTIME140!memcmp` (`mov rax,[rcx]`, 8-byte); 1.5.5 **inlines** the compare into the extension (`mov al,[rcx]` / `cmp al,[rcx+rdx]`, 1-byte), which is exactly why the bucket flips from `VCRUNTIME140.dll!memcmp` to `..._duckdb...pyd!Unknown`. The inlining is a build difference, not a different bug.
- Under the Windows x64 ABI a `memcmp(a, b, n)` takes `rcx=a, rdx=b, r8=n`. `r8` is a plausible length in both (`0x24` = 36, `0x1f00` = 7936), and `rdx` is a sign-extended-negative value in both, with `rbp=0` and `rdi=1` in both.

#### Thread kinds

In 1.5.5 the fault is on the **Python-owned thread**, inside `StreamQueryResult::ExecuteTask` and bottoming out through `python313`. In the 1.4.4 dump it is on **a native thread not owned by CPython** (`ucrtbase!thread_start` -> `kernel32!BaseThreadInitThunk` -> `ntdll!RtlUserThreadStart`, all genuinely resolved symbols). So it faults on both thread kinds. *Inference, not measurement:* that fits shared state being read after free or realloc rather than a bug local to one thread.

#### Ruled out on my side

Each of these was tested by changing it and re-running the identical workload; none of them changed the crash rate in the direction the hypothesis predicts:

- **Not a per-worker memory cap.** Workers die well below the Job Object commit cap (n=24 sampling).
- **Not `memory_limit`.** Raised from 1.5 GB to 5 GB with the process cap at 12288 MB: crashes arrived **sooner and more often** (1.80/min vs 0.59/min). No OOM line, no eviction, no memory-pressure restart anywhere in the pool output.
- **Not `threads`.** Set to `1` machine-wide and verified inside the live processes' environment blocks: all three fresh workers still died within 2 minutes.
- **Not a concurrent writer.** No `.wal` beside any of the attached database files during the window, and no other process had them open.

#### Possibly related, but I do not think it is the same

duckdb/duckdb#25011 is also `0xC0000005` on Windows with `spatial` on 1.5.5, but it is interrupt-triggered under Jupyter and the reporter traced it to EDR hooks and the ipykernel Win32 interrupt event. Mine has no interrupt, no notebook, and no EDR involvement, and it also occurs on 1.4.4. Different mechanism, possibly adjacent code.

### To Reproduce

I do not have a minimal reproducer, and I want to be straight about that rather than post a script that does not fail. What I have is a reliable statistical reproduction of my own workload, plus dumps.

The workload, described generically:

- A FastAPI/uvicorn worker pool, **3 worker processes**, each with its own DuckDB connection.
- Each worker runs **concurrent read-only queries** against a **~1 GB DuckDB file attached read-only**, with the `spatial` extension loaded, while a batch job drives them.
- No writes, no `ATTACH` in RW mode, no `INSTALL` during the run, no interrupts or cancellations.
- Workers are cold-started; the first faults arrive **within seconds to ~90 s** of a worker starting.
- Deaths cluster at two moments: **batch start**, and **the instant clients disconnect**.

Instrumentation that makes it visible (a completed batch is *not* evidence of health — the pool respawns in place and the port stays listening):

```powershell
Get-WinEvent -FilterHashtable @{LogName='Application';Id=1000;StartTime=(Get-Date).AddHours(-6)} |
Where-Object { $_.Message -match 'python' } |
ForEach-Object { $_.TimeCreated, $_.Properties[8].Value, $_.Properties[3].Value, $_.Properties[7].Value }
```

Capturing dumps, in case it helps anyone reproducing on Windows: set `HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\python.exe` with `DumpFolder`, `DumpType`**=1**, `DumpCount`=N. `DumpType=2` (full memory) silently writes **nothing** for job-capped worker processes, because `WerFault` is spawned as a child of the crashing process and inherits its commit cap. A minidump is sufficient for this defect. Analyse with `cdb -z -y srv**https://msdl.microsoft.com/download/symbols -i ` — without `-i` the module cannot be resolved at all.

**What I can supply on request:** three 1.5.5 minidumps (14.8 / 15.7 / 16.9 MB), the exact `.pyd` by hash below, the 1.4.4 minidumps, and full `cdb` transcripts. I have not attached the dumps here because minidumps carry memory fragments from a business workload.

### Binary identity

The Windows wheel's extension binary **carries no version resource** — `FileVersion` and `ProductVersion` are both empty — which is why Windows crash telemetry reports `version: 0.0.0.0` for every one of these crashes. It has to be identified by hash:

```
_duckdb.cp313-win_amd64.pyd 37,374,464 bytes
SHA256 C91833047245D694CD25E146E4F7FB4F412E71B0EAAE257E190A83E8F264A1DF
PE Timestamp 6A5F82FF (Tue Jul 21 10:32:31 2026) ImageSize 023B0000 <- 1.5.5
PE Timestamp 69740402 (Fri Jan 23 18:28:02 2026) ImageSize 021CE000 <- 1.4.4 (different machine)
```

Runtime identity on the 1.5.5 host:

```
duckdb.__version__ = 1.5.5
library_version = v1.5.5
extensions: ('spatial', 'eb1e57c', True) ('json', 'v1.5.5', True)
VCRUNTIME140 14.51.36247.0 (the 1.4.4 host had 14.42.34438.0)
```

### What I am asking for

1. **Symbolicate the module offsets `0x18dc020`, `0x18dc060` and `0x18dc0a0`** (24 / 11 / 1 occurrences in one day) against the official 1.5.5 Windows CPython 3.13 build artefact matching the SHA256 above, and the caller of `VCRUNTIME140!memcmp+0x50` in 1.4.4. Without PDBs every frame name I have is a nearest-export guess, so this is the single step that would turn the report into a diagnosis, and it is the one thing that cannot be done from outside. Publishing PDBs alongside the Windows wheels would let reporters do it themselves.
2. **Ship a version resource in `_duckdb*.pyd`.** `version: 0.0.0.0` in Windows crash telemetry makes fleet-wide triage impossible without hashing every file by hand.
3. **Please treat the 1.4.4 and 1.5.5 reports as one defect** unless symbolication says otherwise — see the section above; bucket-based dedup will split them.

If the engine frames (`ExpressionExecutor`, `StreamQueryResult::ExecuteTask`) make this a core issue rather than a client one, please move or retarget it — I filed here because the crashing artefact is the wheel's binary and asks 1 and 2 are wheel-build concerns.

### OS:

Windows 11 Pro build 22631 (10.0.22631), x86_64, 8 cores, 63.6 GB RAM. Also reproduced on a second Windows x86_64 machine.

### DuckDB Package Version:

1.5.5 (also reproduced on 1.4.4). Latest release tested: **yes, 1.5.5**. Nightly: **not tested**.

### Python Version:

3.13.15 (MSC v.1944, 64-bit) on the 1.5.5 host; 3.13.5 (MSC v.1943, 64-bit) on the 1.4.4 host.

### Full Name:

As on my GitHub profile (this account is my own, under my real first name).

### Affiliation:

Independent — no corporate affiliation.

### Did you include all relevant data sets for reproducing the issue?

No - I cannot share the data sets because they are confidential

### Did you include all code required to reproduce the issue?

No — I have no minimal reproducer, only a statistical reproduction of a private workload plus minidumps and stacks. Stated plainly rather than filling the checkbox.

### Did you include all relevant configuration to reproduce the issue?

Yes, I have

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.