apache / apache/airflow

Implement SecretsMasker for Variables and Connections in the Go SDK

Open
#67,599 1 comment 0 reactions 1 assignee Claimed by @jason810496 View on GitHub
go-sdk kind:feature priority:medium
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Dependency

Wait until both #67317 and #67318 merge. #67317 lands the coordinator-side ``GetVariable`` / ``GetConnection`` TODO sites; #67318 lands the ``pkg/worker/runner.go`` task log handler that the masker plugs into. Until both are on ``main``, the consumer-side hook this feature needs does not exist.

### Background

Python's ``airflow.models.variable.Variable.get`` and ``airflow.models.connection.Connection.get`` register sensitive values with a ``SecretsMasker`` so any subsequent task log line that contains the secret is automatically redacted. The Go SDK currently does not — secrets retrieved via ``GetVariable`` / ``GetConnection`` flow straight into task logs unredacted.

The gap is documented as four coordinated TODOs across the SDK and the runtime. None of them is useful in isolation: log redaction needs both a producer (the lookup site registering the secret) and a consumer (the log handler running the redaction). Until all four land, the SDK ships without log-side masking parity.

HTTP-backed client (already on ``main``):

```go
// go-sdk/sdk/client.go - GetVariable
// TODO: register secret-named variables with a SecretsMasker so the
// returned value is automatically redacted from subsequent task logs,
// matching Python's airflow.models.variable.Variable.get behaviour.
// Pairs with the "TODO: mask secrets here" hook in
// pkg/worker/runner.go's task log handler — both halves are needed
// before secret masking actually works end-to-end.
```

```go
// go-sdk/sdk/client.go - GetConnection
// TODO: register Connection.Password and sensitive-keyed entries of
// Connection.Extra with a SecretsMasker so they are auto-redacted from
// subsequent task logs, matching Python's
// airflow.models.connection.Connection.get behaviour. Pairs with the
// "TODO: mask secrets here" hook in pkg/worker/runner.go's task log
// handler and the matching TODO on GetVariable above.
```

Coordinator-mode client (lands with #67317):

```go
// go-sdk/pkg/execution/client.go - GetVariable and GetConnection
// (same TODO text, same SecretsMasker hook)
```

Log-handler hook (lands with #67318): the ``// TODO: mask secrets here`` comment in ``go-sdk/pkg/worker/runner.go`` is where the consumer side of the redaction belongs.

### What needs to happen

1. Design and add a ``SecretsMasker`` type to the Go SDK with a register-and-redact API roughly equivalent to ``airflow.utils.log.secrets_masker.SecretsMasker`` (case-insensitive secret-name detection, replace-with-``***`` semantics, per-task scoping).
2. Register secret-named variables in both ``sdk.client.GetVariable`` and ``execution.CoordinatorClient.GetVariable``.
3. Register ``Connection.Password`` and sensitive-keyed entries of ``Connection.Extra`` in both ``sdk.client.GetConnection`` and ``execution.CoordinatorClient.GetConnection``.
4. Wire the masker into the task log handler at the ``// TODO: mask secrets here`` site in ``pkg/worker/runner.go`` so emitted log lines are redacted before they leave the runtime.
5. Remove the four coordinated TODOs above.

### Acceptance criteria

- Looking up a variable named e.g. ``my_password`` via either ``sdk.client`` or ``CoordinatorClient`` causes any subsequent task log line containing the variable's value to be redacted to ``***`` (or the project's chosen sentinel).
- ``Connection.Password`` and sensitive-keyed ``Connection.Extra`` entries produce the same redaction behaviour from both clients.
- Tests cover: (a) registration from each client, (b) redaction in the log handler, (c) negative case where a non-sensitive variable name is not registered.
- All four TODOs cross-referencing this work are removed from the codebase.

### Context

- Originating review: self-review thread on #67317 (coordinator comms + client PR).
- HTTP-side TODOs (already on ``main``): ``go-sdk/sdk/client.go`` ``GetVariable`` and ``GetConnection``.
- Coordinator-side TODOs (land with #67317): ``go-sdk/pkg/execution/client.go`` ``GetVariable`` and ``GetConnection``.
- Log-handler hook (lands with #67318): ``go-sdk/pkg/worker/runner.go``.
- Python parity: ``airflow.utils.log.secrets_masker.SecretsMasker``, ``airflow.models.variable.Variable.get``, ``airflow.models.connection.Connection.get``.
- Related project board: https://github.com/orgs/apache/projects/499/views/1

---
Drafted-by: Claude Code (Opus 4.7) (no human review before posting)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.