duckdb / duckdb/duckdb-python

DuckDB spawns worker threads on module import rather than connection creation

Abierto
#612 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
187
Forks
112
Merge medio
13 h 29 min
PR fusionados (30 d)
17

Descripción

### What happens?

Importing the duckdb Python module (`import duckdb`) immediately spawns ~100 worker threads (appears to be ncpus), even without creating any connections. This causes unexpected resource consumption in applications that import the module but may not immediately use it.

## Expected Behavior
Worker threads should only be created when:
1. An actual database connection is established via `duckdb.connect()`
2. Or when explicitly configured to do so

The module import itself should be lightweight and not spawn background threads.

## Proposed Solution
Consider lazy initialization of the thread pool:
- Defer TaskScheduler/thread pool creation until first `connect()` call
- Or provide an environment variable/config option to control thread spawning behavior on import
- Similar to how other database drivers handle connection pooling

## Workaround
Currently requires monkey-patching or lazy imports to avoid the thread spawning:
```python
# Must be done before any code imports duckdb
import sys
import importlib.util

def lazy_import_duckdb():
spec = importlib.util.find_spec("duckdb")
module = importlib.util.module_from_spec(spec)
sys.modules["duckdb"] = module
# Don't execute the module yet
return module
```

### To Reproduce

```python
import duckdb # This alone spawns ~100 threads
```

When attaching gdb to a process that has imported duckdb:
- `thread apply all bt` shows 100+ threads with duckdb in their stack traces
- All threads show `ExecuteForever` in their call stacks
- Base stack frame is `clone3` syscall

### OS:

Ubuntu 22.04 x86_64

### DuckDB Version:

0.6.1

### DuckDB Client:

Python

### Hardware:

128 core CPU

### Full Name:

Nova DasSarma

### Affiliation:

Anthropic, PBC

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

- [x] Yes, I have

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Empieza reproduciendo el problema con `import duckdb` en la configuración de Ubuntu indicada e inspecciona la ruta de importación y la inicialización de `duckdb.connect()`. Usa el backtrace de gdb proporcionado, especialmente `ExecuteForever`, para rastrear cuándo se crea el grupo de workers. Se considera terminado cuando importar el módulo por sí solo no genera los hilos de worker, mientras que crear una conexión sigue inicializando los workers necesarios.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
databases
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
48/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.