dlt-hub / dlt-hub/dlt

qol: `pipeline.run()` should return extract, normalize, load stats

Open
#3,371 1 comment 0 reactions 0 assignees View on GitHub
2.0 breaking QoL
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

This is the canonical dlt code
```python
resources = [...]
pipeline = dlt.pipeline("name", destination=...)
load_info = pipeline.run(resources)
print(load_info)
```

This only returns the information of the Load step (the fastest step), which includes a "time to run this step". It's easy to misinterpret this value as "the time to run the full pipeline".

## How it works
Internally, `pipeline.run()` simply calls `.extract()`, `.normalize()`, and `.load()` which each return metadata. Only the last `.load()` is returned by `.run()`
```python
class Pipeline:
def run(self, ...):
# ...
# extract from the source
if data is not None:
self.extract(
data,
table_name=table_name,
write_disposition=write_disposition,
columns=columns,
primary_key=primary_key,
schema=schema,
table_format=table_format,
schema_contract=schema_contract,
refresh=refresh or self.refresh,
loader_file_format=loader_file_format,
)
self.normalize()
return self.load(destination, dataset_name, credentials=credentials)
else:
return None
```

## Proposed changes
`.run()` could return a list of those or a new "aggregate object". This would be a breaking change and should be in `dlt>=2.0.0`

Contributor guide

Open the contributing guide

Research direction

Start with the Pipeline.run entry point described in the issue, then trace the extract(), normalize(), and load() metadata it currently invokes. Decide whether dlt>=2.0 should return a list or an aggregate object, and define how the combined extract, normalize, and load statistics should be represented without confusing step time with full-pipeline time.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.