apache / apache/airflow

Support Taskflow-like syntax in the Go SDK

Open
#66,937 1 comment 0 reactions 1 assignee Claimed by @jason810496 View on GitHub
go-sdk kind:feature priority:high
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Background

Today a Go task function takes `ctx`, an `sdk.Client` / `sdk.VariableClient`,
and a `*slog.Logger`, and pulls XCom data explicitly:

```go
// go-sdk/example/bundle/main.go
func transform(ctx context.Context, client sdk.VariableClient, log *slog.Logger) error {
val, err := client.GetVariable(ctx, "my_variable")
...
}
```

Java has just landed Taskflow-style task signatures under #66332, where
the upstream task's return value is declaratively injected into the
downstream task's parameter list rather than pulled via the client. We
want the same shape in Go so authors can write:

```go
func extract(ctx context.Context) (SomeType, error) { ... }
func transform(ctx context.Context, upstream SomeType) error { ... }
```

### What needs to happen

1. Decide how to declare task-to-parameter wiring in `Registry.AddTask`
(does `depends []string` carry positional binding, or do we use typed
task handles returned by `AddTask`?).
2. Use reflection in `bundlev1.NewTaskFunction` to recognise an extra
typed parameter and treat it as an XCom pull from the named upstream.
3. Serialise the binding into the Dag JSON in coordinator-mode so the
supervisor knows which XCom to feed in.
4. Update `example/bundle/main.go` to a Taskflow-style example.

### Acceptance criteria

- A Go task can declare an upstream-typed parameter and receive the
upstream's return value at execution time without calling
`client.GetXCom` explicitly.
- Serialisation matches what the Python supervisor expects for a
Taskflow-wired Dag (validated against `test_dags.yaml`).
- The existing client-pull pattern remains supported.

### Context

- Java parallel: #66332 (Done).
- `@task.stub` argument-passing groundwork: #66330 (Done).
- Same area as task dependency representation: #66331.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.