Azure / Azure/Azure-DataFactory
Mapping Data Flow: Include source metrics (rowsread) as well as sink metrics in output metrics data
- Dominant language
- PowerShell
- Stars
- 529
- Forks
- 623
- PR merge metrics
- No merged PRs in 30d
Description
When using mapping data flows it impossible to pull out metrics for the number of rowsread from a source to include in logging, without a reall clunky work around.
In the parent adf pipeline its possible to reference the metrics of the sinks in the data flow task and capture the rows written (separate sinks for inserts/updates/deletes)
_@activity('Populate DMStaging').output.runStatus.metrics.DMStagingDeleted.rowsWritten_
However i'm not sure why the sources can't also be included in the metrics section of the output details. This would make things a lot simpler than clunky workarounds of having to create multiple branches and aggregate values then writing these out to dummy files etc for what is a pretty standard requirement for most ETL pipelines ie know how many records you extracted and be able to compare that with how many got inserted updated or didn't change. Having these metrics referable or the ability to assign them to a suitable variable that can be referenced is a bit of an oversight i feel.

Contributor guide
Assessment
This issue has not been assessed yet.