dlt-hub / dlt-hub/dlt

[Bug] Parallel extraction for large MongoDB collections (Intra-resource parallelism)

Open
#3,583 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### dlt version

1.21.0

### Describe the problem

I am currently running an incremental pipeline extracting data from **MongoDB to Snowflake**, using arrow data item format. My collections are massive (exceeding **100 million rows**).

Currently, the extraction throughput is capped at approximately **1100 rows/second**. At this speed, initial loads and large incremental updates take an impractical amount of time.

I attempted to speed this up by enabling parallelism on the resource:
```python
# Pseudo-code of attempt
mongodb_collection(
parallel=True,
# ... logic ...
```

**Result:** The pipeline simply hangs/freezes and does not extract anything. It appears that `dlt` currently supports parallel execution *between* different resources, but not *within* a single resource (splitting one collection into multiple parallel extraction threads).

### Describe the solution you'd like
I would like native support for **intra-resource parallelism** for the MongoDB source.

Ideally, the MongoDB source could automatically (or via configuration) chunk the extraction based on:
1. MongoDB `_id` ranges.
2. Or a specific indexed field (e.g., `updated_at` or a numeric ID).

This would allow multiple threads/processes to extract from the *same* large collection simultaneously, significantly increasing throughput for massive datasets.

### Additional context
* **Source:** MongoDB
* **Destination:** Snowflake
* **Data Volume:** > 100 Million rows
* **Current Speed:** ~1100 rows/sec
* **Behavior when `parallel=True`:** Process hangs indefinitely.

---

### Expected behavior

I would imagine it would chunk the extraction into multiple threads.

### Steps to reproduce

@dlt.source(
name="mongodb_replication_append_only_source", max_table_nesting=0
) # max_table_nesting wont work with data_item_format arrow, only with object
def mongodb_replication_append_only_source():

new_rows_resource = mongodb_collection(
connection_url=dlt.secrets.get("sources.mongodb.connection_url"),
database=MONGO_DATABASE_NAME,
collection=table_config["original_table_name"],
incremental=dlt.sources.incremental(
cursor_path=table_config["original_table_incremental_column"],
on_cursor_value_missing="exclude",
),
write_disposition="merge",
data_item_format="arrow",
chunk_size=int(os.environ["DLT_CHUNK_SIZE"]),
parallel=True,
)
new_rows_resource.write_disposition = "merge"

new_rows_resource.apply_hints(
table_name=table_config["original_table_name"].upper(),
primary_key="_id",
)

yield new_rows_resource

def main():
pipeline_name = str(os.getenv("DLT_PIPELINE_NAME"))
dataset_name = str(os.getenv("DLT_DESTINATION_DATASET_NAME"))
pipeline_refresh = os.getenv("DLT_PIPELINE_REFRESH")

if pipeline_refresh in {"drop_sources", "drop_resources", "drop_data"}:
pass
elif pipeline_refresh == "None" or pipeline_refresh is None:
pipeline_refresh = None
else:
raise ValueError(f"Invalid Refresh: {pipeline_refresh}")

dlt.config["destination.snowflake.keep_staged_files"] = False

pipeline = dlt.pipeline(
pipeline_name=pipeline_name,
destination="snowflake",
staging="filesystem",
dataset_name=dataset_name,
refresh=pipeline_refresh,
progress="log",
)

dlt.config[f"{pipeline_name}.data_writer.file_max_bytes"] = 128 * 1024 * 1024
dlt.config["load.delete_completed_jobs"] = True
dlt.config["load.workers"] = int(os.getenv("DLT_LOAD_WORKERS", "20"))
dlt.config["extract.workers"] = 10

pipeline.sync_destination()

load_info = pipeline.run(
data=mongodb_replication_append_only_source(),
loader_file_format="parquet",
)
logger.info(pipeline.last_trace)
load_info.raise_on_failed_jobs()

if __name__ == "__main__":
main()

### Operating system

Linux

### Runtime environment

Docker, Docker Compose

### Python version

3.13

### dlt data source

mongo verified source, with some fixes to make arrow data format work, since the original in the repository wont work as is.

### dlt destination

Snowflake

### Other deployment details

_No response_

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the mongo verified source's mongodb_collection entry point and reproduce the provided single-resource parallel=True case with the stated environment. Done means a large collection can be split and extracted concurrently without hanging, with the requested chunking or configuration behavior documented or tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.