apache / apache/airflow

Record mapped_length for Java SDK tasks with mapped dependants

Open
#71,061 0 comments 0 reactions 1 assignee Claimed by @jason810496 View on GitHub
area:core area:dynamic-task-mapping area:java-sdk kind:feature on hold
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Background

A Java task's return value cannot currently feed a downstream `.expand()`. The Python task runner does this in `_push_xcom_if_needed`, but a foreign runtime cannot inspect the Dag to learn that its output has mapped dependants.

https://github.com/apache/airflow/pull/70571 introduces the server-derived `TIRunContext.has_mapped_dependants` flag (computed from `iter_mapped_dependants`); when set, the supervisor records `mapped_length = len(value)` on the return-value `SetXCom` on the task's behalf, so the scheduler can expand the dependants.

### What needs to happen

1. Confirm whether the supervisor-side recording from https://github.com/apache/airflow/pull/70571 is language-agnostic. If it is, this issue reduces to Java-side verification plus tests; if any of it is Go-specific, lift it into the shared coordinator/supervisor path.
2. Ensure the Java SDK surfaces `has_mapped_dependants` on its context object where it is useful to task authors.
3. Cover the flow end to end: a Java `@task.stub` returning a list, with a downstream Python `.expand()` over it.

### Acceptance criteria

- A Java stub task whose return value feeds a downstream `.expand()` causes those dependants to expand to `len(value)` instances.
- A Java task with no mapped dependants records no `mapped_length` (no behaviour change).
- The version gate is respected: pre-arg-bindings clients skip the derivation entirely.

### Context

- Depends on https://github.com/apache/airflow/pull/70570 and https://github.com/apache/airflow/pull/70571
- Python reference: `_push_xcom_if_needed` in the task-SDK task runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.