duckdb / duckdb/duckdb-python

DuckDB spawns worker threads on module import rather than connection creation

Aperta
#612 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
187
Fork
112
Merge medio
13h 29m
PR unite (30g)
17

Descrizione

### What happens?

Importing the duckdb Python module (`import duckdb`) immediately spawns ~100 worker threads (appears to be ncpus), even without creating any connections. This causes unexpected resource consumption in applications that import the module but may not immediately use it.

## Expected Behavior
Worker threads should only be created when:
1. An actual database connection is established via `duckdb.connect()`
2. Or when explicitly configured to do so

The module import itself should be lightweight and not spawn background threads.

## Proposed Solution
Consider lazy initialization of the thread pool:
- Defer TaskScheduler/thread pool creation until first `connect()` call
- Or provide an environment variable/config option to control thread spawning behavior on import
- Similar to how other database drivers handle connection pooling

## Workaround
Currently requires monkey-patching or lazy imports to avoid the thread spawning:
```python
# Must be done before any code imports duckdb
import sys
import importlib.util

def lazy_import_duckdb():
spec = importlib.util.find_spec("duckdb")
module = importlib.util.module_from_spec(spec)
sys.modules["duckdb"] = module
# Don't execute the module yet
return module
```

### To Reproduce

```python
import duckdb # This alone spawns ~100 threads
```

When attaching gdb to a process that has imported duckdb:
- `thread apply all bt` shows 100+ threads with duckdb in their stack traces
- All threads show `ExecuteForever` in their call stacks
- Base stack frame is `clone3` syscall

### OS:

Ubuntu 22.04 x86_64

### DuckDB Version:

0.6.1

### DuckDB Client:

Python

### Hardware:

128 core CPU

### Full Name:

Nova DasSarma

### Affiliation:

Anthropic, PBC

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

- [x] Yes, I have

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia riproducendo il problema con `import duckdb` sulla configurazione Ubuntu segnalata e analizza il percorso di importazione e l’inizializzazione di `duckdb.connect()`. Usa il backtrace di gdb fornito, in particolare `ExecuteForever`, per tracciare quando viene creato il pool di worker. Il lavoro è completato quando la sola importazione del modulo non avvia i thread worker, mentre la creazione di una connessione continua a inizializzare i worker necessari.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
databases
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.