DuckDB spawns worker threads on module import rather than connection creation
- Lenguaje dominante
- Python
- Estrellas
- 187
- Forks
- 112
- Merge medio
- 13 h 29 min
- PR fusionados (30 d)
- 17
Descripción
### What happens?
Importing the duckdb Python module (`import duckdb`) immediately spawns ~100 worker threads (appears to be ncpus), even without creating any connections. This causes unexpected resource consumption in applications that import the module but may not immediately use it.
## Expected Behavior
Worker threads should only be created when:
1. An actual database connection is established via `duckdb.connect()`
2. Or when explicitly configured to do so
The module import itself should be lightweight and not spawn background threads.
## Proposed Solution
Consider lazy initialization of the thread pool:
- Defer TaskScheduler/thread pool creation until first `connect()` call
- Or provide an environment variable/config option to control thread spawning behavior on import
- Similar to how other database drivers handle connection pooling
## Workaround
Currently requires monkey-patching or lazy imports to avoid the thread spawning:
```python
# Must be done before any code imports duckdb
import sys
import importlib.util
def lazy_import_duckdb():
spec = importlib.util.find_spec("duckdb")
module = importlib.util.module_from_spec(spec)
sys.modules["duckdb"] = module
# Don't execute the module yet
return module
```
### To Reproduce
```python
import duckdb # This alone spawns ~100 threads
```
When attaching gdb to a process that has imported duckdb:
- `thread apply all bt` shows 100+ threads with duckdb in their stack traces
- All threads show `ExecuteForever` in their call stacks
- Base stack frame is `clone3` syscall
### OS:
Ubuntu 22.04 x86_64
### DuckDB Version:
0.6.1
### DuckDB Client:
Python
### Hardware:
128 core CPU
### Full Name:
Nova DasSarma
### Affiliation:
Anthropic, PBC
### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
### Did you include all relevant data sets for reproducing the issue?
Not applicable - the reproduction does not require a data set
### Did you include all code required to reproduce the issue?
- [x] Yes, I have
### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?
- [x] Yes, I have
Guía de contribución
Línea de trabajo
Empieza reproduciendo el problema con `import duckdb` en la configuración de Ubuntu indicada e inspecciona la ruta de importación y la inicialización de `duckdb.connect()`. Usa el backtrace de gdb proporcionado, especialmente `ExecuteForever`, para rastrear cuándo se crea el grupo de workers. Se considera terminado cuando importar el módulo por sí solo no genera los hilos de worker, mientras que crear una conexión sigue inicializando los workers necesarios.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- python
- Área
- databases
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Tranquilo
- Claridad
- Bastante claro
- Aptitud para principiantes
- 48/100