feat(qol): toggle multiprocessing / multithreading
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
By default, `dlt` uses multiprocessing to efficiently process data (especially `normalize` and `load` steps). However, it makes debugging harder (longer stack trace, loss of context between caller and process pool, etc.)
## Request
Have an option to entirely disable multiprocessing for debugging. Performance wouldn't be the priority (could emit a warning about it)
This could be configure via `config.toml`
Contributor guide
Research direction
Start by tracing how multiprocessing is used in the normalize and load steps, then inspect how config.toml options are handled. Done means a configuration option can disable multiprocessing for debugging, with the requested behavior applying to those processing steps and any performance warning documented or emitted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100