duckdb / duckdb/duckdb-python

DuckDB spawns worker threads on module import rather than connection creation

Open
#612 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
187
Forks
112
Avg merge
13h 29m
Merged PRs (30d)
17

Description

### What happens?

Importing the duckdb Python module (`import duckdb`) immediately spawns ~100 worker threads (appears to be ncpus), even without creating any connections. This causes unexpected resource consumption in applications that import the module but may not immediately use it.

## Expected Behavior
Worker threads should only be created when:
1. An actual database connection is established via `duckdb.connect()`
2. Or when explicitly configured to do so

The module import itself should be lightweight and not spawn background threads.

## Proposed Solution
Consider lazy initialization of the thread pool:
- Defer TaskScheduler/thread pool creation until first `connect()` call
- Or provide an environment variable/config option to control thread spawning behavior on import
- Similar to how other database drivers handle connection pooling

## Workaround
Currently requires monkey-patching or lazy imports to avoid the thread spawning:
```python
# Must be done before any code imports duckdb
import sys
import importlib.util

def lazy_import_duckdb():
spec = importlib.util.find_spec("duckdb")
module = importlib.util.module_from_spec(spec)
sys.modules["duckdb"] = module
# Don't execute the module yet
return module
```

### To Reproduce

```python
import duckdb # This alone spawns ~100 threads
```

When attaching gdb to a process that has imported duckdb:
- `thread apply all bt` shows 100+ threads with duckdb in their stack traces
- All threads show `ExecuteForever` in their call stacks
- Base stack frame is `clone3` syscall

### OS:

Ubuntu 22.04 x86_64

### DuckDB Version:

0.6.1

### DuckDB Client:

Python

### Hardware:

128 core CPU

### Full Name:

Nova DasSarma

### Affiliation:

Anthropic, PBC

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

- [x] Yes, I have

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.