[Bug] Parallel extraction for large MongoDB collections (Intra-resource parallelism)
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 605
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### dlt version
1.21.0
### Describe the problem
I am currently running an incremental pipeline extracting data from **MongoDB to Snowflake**, using arrow data item format. My collections are massive (exceeding **100 million rows**).
Currently, the extraction throughput is capped at approximately **1100 rows/second**. At this speed, initial loads and large incremental updates take an impractical amount of time.
I attempted to speed this up by enabling parallelism on the resource:
```python
# Pseudo-code of attempt
mongodb_collection(
parallel=True,
# ... logic ...
```
**Result:** The pipeline simply hangs/freezes and does not extract anything. It appears that `dlt` currently supports parallel execution *between* different resources, but not *within* a single resource (splitting one collection into multiple parallel extraction threads).
### Describe the solution you'd like
I would like native support for **intra-resource parallelism** for the MongoDB source.
Ideally, the MongoDB source could automatically (or via configuration) chunk the extraction based on:
1. MongoDB `_id` ranges.
2. Or a specific indexed field (e.g., `updated_at` or a numeric ID).
This would allow multiple threads/processes to extract from the *same* large collection simultaneously, significantly increasing throughput for massive datasets.
### Additional context
* **Source:** MongoDB
* **Destination:** Snowflake
* **Data Volume:** > 100 Million rows
* **Current Speed:** ~1100 rows/sec
* **Behavior when `parallel=True`:** Process hangs indefinitely.
---
### Expected behavior
I would imagine it would chunk the extraction into multiple threads.
### Steps to reproduce
@dlt.source(
name="mongodb_replication_append_only_source", max_table_nesting=0
) # max_table_nesting wont work with data_item_format arrow, only with object
def mongodb_replication_append_only_source():
new_rows_resource = mongodb_collection(
connection_url=dlt.secrets.get("sources.mongodb.connection_url"),
database=MONGO_DATABASE_NAME,
collection=table_config["original_table_name"],
incremental=dlt.sources.incremental(
cursor_path=table_config["original_table_incremental_column"],
on_cursor_value_missing="exclude",
),
write_disposition="merge",
data_item_format="arrow",
chunk_size=int(os.environ["DLT_CHUNK_SIZE"]),
parallel=True,
)
new_rows_resource.write_disposition = "merge"
new_rows_resource.apply_hints(
table_name=table_config["original_table_name"].upper(),
primary_key="_id",
)
yield new_rows_resource
def main():
pipeline_name = str(os.getenv("DLT_PIPELINE_NAME"))
dataset_name = str(os.getenv("DLT_DESTINATION_DATASET_NAME"))
pipeline_refresh = os.getenv("DLT_PIPELINE_REFRESH")
if pipeline_refresh in {"drop_sources", "drop_resources", "drop_data"}:
pass
elif pipeline_refresh == "None" or pipeline_refresh is None:
pipeline_refresh = None
else:
raise ValueError(f"Invalid Refresh: {pipeline_refresh}")
dlt.config["destination.snowflake.keep_staged_files"] = False
pipeline = dlt.pipeline(
pipeline_name=pipeline_name,
destination="snowflake",
staging="filesystem",
dataset_name=dataset_name,
refresh=pipeline_refresh,
progress="log",
)
dlt.config[f"{pipeline_name}.data_writer.file_max_bytes"] = 128 * 1024 * 1024
dlt.config["load.delete_completed_jobs"] = True
dlt.config["load.workers"] = int(os.getenv("DLT_LOAD_WORKERS", "20"))
dlt.config["extract.workers"] = 10
pipeline.sync_destination()
load_info = pipeline.run(
data=mongodb_replication_append_only_source(),
loader_file_format="parquet",
)
logger.info(pipeline.last_trace)
load_info.raise_on_failed_jobs()
if __name__ == "__main__":
main()
### Operating system
Linux
### Runtime environment
Docker, Docker Compose
### Python version
3.13
### dlt data source
mongo verified source, with some fixes to make arrow data format work, since the original in the repository wont work as is.
### dlt destination
Snowflake
### Other deployment details
_No response_
### Additional information
_No response_
Contributor guide
Research direction
Start at the mongo verified source's mongodb_collection entry point and reproduce the provided single-resource parallel=True case with the stated environment. Done means a large collection can be split and extracted concurrently without hanging, with the requested chunking or configuration behavior documented or tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100