dlt-hub / dlt-hub/dlt

feat(qol): toggle multiprocessing / multithreading

Open
#3,576 2 comments 0 reactions 0 assignees View on GitHub
QoL
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

By default, `dlt` uses multiprocessing to efficiently process data (especially `normalize` and `load` steps). However, it makes debugging harder (longer stack trace, loss of context between caller and process pool, etc.)

## Request
Have an option to entirely disable multiprocessing for debugging. Performance wouldn't be the priority (could emit a warning about it)

This could be configure via `config.toml`

Contributor guide

Open the contributing guide

Research direction

Start by tracing how multiprocessing is used in the normalize and load steps, then inspect how config.toml options are handled. Done means a configuration option can disable multiprocessing for debugging, with the requested behavior applying to those processing steps and any performance warning documented or emitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.