Azure / Azure/azure-sdk-for-python

azure-mgmt-datafactory: ExecuteDataFlowActivity drops the data-flow reference (model maps 'dataFlow' but the service uses 'dataflow')

Open
#48,704 1 comment 1 reaction 0 assignees View on GitHub
customer-reported Data Factory Mgmt needs-team-triage question
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

### Library name and version

azure-mgmt-datafactory 9.2.0 and 10.0.0 (latest)

### Describe the bug

An `ExecuteDataFlowActivity`'s data-flow reference is silently lost when a pipeline is read (or round-tripped) through the SDK model. The service serializes the reference as lowercase `typeProperties.dataflow`, but the generated model maps the `data_flow` attribute to `typeProperties.dataFlow` (capital F). On deserialization the lowercase key is unmatched, `data_flow` becomes `None`, and it disappears from `serialize()` — so `pipelines.get(...).serialize()` returns the activity without its data-flow reference, and any edit written back via `pipelines.create_or_update(...)` drops it entirely, turning a working Data Flow activity into a no-op.

Its sibling `typeProperties` fields (`compute`, `staging`, `traceLevel`) survive, because their casing matches the model — which makes the loss easy to miss.

**Casing, per Microsoft's own docs:** the [Data Flow activity reference](https://learn.microsoft.com/en-us/azure/data-factory/control-flow-execute-data-flow-activity#syntax) documents the property as lowercase `dataflow` (and the Type-properties table lists `dataflow` as Required). The model, however:

```
>>> from azure.mgmt.datafactory.models import ExecuteDataFlowActivity
>>> ExecuteDataFlowActivity._attribute_map["data_flow"]
{'key': 'typeProperties.dataFlow', 'type': 'DataFlowReference'}
```

In 10.0.0 the same mismatch is present (`data_flow: DataFlowReference = rest_field(name="dataFlow")`).

### Expected behavior

`pipelines.get(...)` should populate `data_flow`, and `serialize()` / `create_or_update(...)` should round-trip the reference. i.e. the model key should match what the service emits (`dataflow`).

### Actual behavior

`data_flow` deserializes to `None`; the reference is absent from `serialize()`; a round-tripped pipeline loses it.

### Reproduction Steps

No Azure resources or credentials needed — the loss is at the model boundary:

```python
from azure.mgmt.datafactory.models import Activity, ExecuteDataFlowActivity

print(ExecuteDataFlowActivity._attribute_map["data_flow"])
# {'key': 'typeProperties.dataFlow', 'type': 'DataFlowReference'}

# 'dataflow' (lowercase) is exactly what the service returns for this activity
act = Activity.deserialize({
"name": "df",
"type": "ExecuteDataFlow",
"typeProperties": {
"dataflow": {"referenceName": "MyFlow", "type": "DataFlowReference"},
"compute": {"computeType": "General", "coreCount": 8},
"traceLevel": "Fine",
},
})

print(type(act).__name__) # ExecuteDataFlowActivity
print(act.data_flow) # None <-- reference lost
print(act.serialize()["typeProperties"])
# {'compute': {'computeType': 'General', 'coreCount': 8}, 'traceLevel': 'Fine'}
# ^ no 'dataflow'/'dataFlow' — compute & traceLevel survived, the reference did not
```

Confirmed live against a real ADF factory: a pipeline whose Data Flow activity executes correctly returns via `pipelines.get(...)` with no data-flow reference in the serialized output; the raw ARM REST response for the same pipeline contains `typeProperties.dataflow` (lowercase).

### Prior reports

This has been reported and closed several times without a fix, most recently auto-closed by the stale bot (which invites a fresh issue):
- #23488 ("dataflow Reference is missing in Pipeline.get")
- #32382 (closed *not planned*)
- #35832 (auto-closed 2026-06-01: "please feel free to create a new issue, referencing this one")

### Suggested fix

Change the generated key for `ExecuteDataFlowActivity.data_flow` from `dataFlow` to `dataflow` (in the TypeSpec/Swagger the SDK is generated from), so it matches what the ADF service actually emits and accepts. If the service genuinely also accepts `dataFlow`, the mismatch is still worth resolving in the spec so `get`→`serialize`/`create_or_update` round-trips are lossless.

Contributor guide

Open the contributing guide

Research direction

Start with the provided Python reproduction and inspect ExecuteDataFlowActivity._attribute_map, Activity.deserialize, and serialize. Done means the service's lowercase typeProperties.dataflow populates data_flow and survives serialization and the pipelines get/create_or_update round trip without Azure credentials.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
api, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.