Azure / Azure/Azure-DataFactory

Mapping Data Flow: Include source metrics (rowsread) as well as sink metrics in output metrics data

Open
#196 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PowerShell
Stars
529
Forks
623
PR merge metrics
No merged PRs in 30d

Description

When using mapping data flows it impossible to pull out metrics for the number of rowsread from a source to include in logging, without a reall clunky work around.

In the parent adf pipeline its possible to reference the metrics of the sinks in the data flow task and capture the rows written (separate sinks for inserts/updates/deletes)

_@activity('Populate DMStaging').output.runStatus.metrics.DMStagingDeleted.rowsWritten_

However i'm not sure why the sources can't also be included in the metrics section of the output details. This would make things a lot simpler than clunky workarounds of having to create multiple branches and aggregate values then writing these out to dummy files etc for what is a pretty standard requirement for most ETL pipelines ie know how many records you extracted and be able to compare that with how many got inserted updated or didn't change. Having these metrics referable or the ability to assign them to a suitable variable that can be referenced is a bit of an oversight i feel.

![image](https://user-images.githubusercontent.com/41288088/84182110-cd4b7900-aa81-11ea-9a13-a1086dac10b8.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.