qol: `pipeline.run()` should return extract, normalize, load stats
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
This is the canonical dlt code
```python
resources = [...]
pipeline = dlt.pipeline("name", destination=...)
load_info = pipeline.run(resources)
print(load_info)
```
This only returns the information of the Load step (the fastest step), which includes a "time to run this step". It's easy to misinterpret this value as "the time to run the full pipeline".
## How it works
Internally, `pipeline.run()` simply calls `.extract()`, `.normalize()`, and `.load()` which each return metadata. Only the last `.load()` is returned by `.run()`
```python
class Pipeline:
def run(self, ...):
# ...
# extract from the source
if data is not None:
self.extract(
data,
table_name=table_name,
write_disposition=write_disposition,
columns=columns,
primary_key=primary_key,
schema=schema,
table_format=table_format,
schema_contract=schema_contract,
refresh=refresh or self.refresh,
loader_file_format=loader_file_format,
)
self.normalize()
return self.load(destination, dataset_name, credentials=credentials)
else:
return None
```
## Proposed changes
`.run()` could return a list of those or a new "aggregate object". This would be a breaking change and should be in `dlt>=2.0.0`
Contributor guide
Research direction
Start with the Pipeline.run entry point described in the issue, then trace the extract(), normalize(), and load() metadata it currently invokes. Decide whether dlt>=2.0 should return a list or an aggregate object, and define how the combined extract, normalize, and load statistics should be represented without confusing step time with full-pipeline time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100