apache / apache/airflow

Bind mapped stub-task arguments in the Java SDK runtime

Open
#71,060 0 comments 0 reactions 1 assignee Claimed by @jason810496 View on GitHub
area:core-operators area:dynamic-task-mapping area:java-sdk kind:feature on hold
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Background

https://github.com/apache/airflow/pull/70570 makes the execution API derive a **per-map-index** arg-binding spec for a dynamically mapped `@task.stub`, so a foreign runtime receives *its* element rather than the aggregated upstream output. https://github.com/apache/airflow/pull/70571 implements the Go-SDK consumer; the Java SDK has no equivalent.

`XComArgBinding` carries two selectors the unmapped path never sets:

- `map_index` — pull the upstream XCom row at that index (expand over a *mapped* upstream).
- `element_index` — pull the unmapped upstream row, then take element N of the resulting sequence (expand over an *unmapped* upstream's list output).

Literal expands are resolved to their element server-side and arrive as plain literals, so the binding surface itself is unchanged from the unmapped case.

### What needs to happen

1. Extend the Java SDK's arg-binding resolution to honour `map_index` and `element_index`, with typed and out-of-range errors that fail the task before its body runs.
2. Keep the unmapped path unchanged: a binding with neither selector takes the whole value.
3. Add an example mapped Dag to the Java example bundle and cover it with an E2E test.

### Acceptance criteria

- A `@task.stub` expanded with `.expand()` delivers the correct per-index element to the Java task for all three expand forms (literal collection, unmapped upstream's list output, mapped upstream).
- Multiple expanded kwargs (the cross product) bind correctly, including mixed expand sources in one call.
- `.partial()` literals and unmapped-upstream XComs bind identically on every index.
- Out-of-range or type-mismatched selectors fail the task loudly rather than silently zero-filling.

### Context

- Depends on https://github.com/apache/airflow/pull/69757 and https://github.com/apache/airflow/pull/70570
- Go parallel: https://github.com/apache/airflow/pull/70571
- Java-side arg binding for unmapped stub calls: https://github.com/apache/airflow/pull/71057

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.