Implement SecretsMasker for Variables and Connections in the Go SDK
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Dependency
Wait until both #67317 and #67318 merge. #67317 lands the coordinator-side ``GetVariable`` / ``GetConnection`` TODO sites; #67318 lands the ``pkg/worker/runner.go`` task log handler that the masker plugs into. Until both are on ``main``, the consumer-side hook this feature needs does not exist.
### Background
Python's ``airflow.models.variable.Variable.get`` and ``airflow.models.connection.Connection.get`` register sensitive values with a ``SecretsMasker`` so any subsequent task log line that contains the secret is automatically redacted. The Go SDK currently does not — secrets retrieved via ``GetVariable`` / ``GetConnection`` flow straight into task logs unredacted.
The gap is documented as four coordinated TODOs across the SDK and the runtime. None of them is useful in isolation: log redaction needs both a producer (the lookup site registering the secret) and a consumer (the log handler running the redaction). Until all four land, the SDK ships without log-side masking parity.
HTTP-backed client (already on ``main``):
```go
// go-sdk/sdk/client.go - GetVariable
// TODO: register secret-named variables with a SecretsMasker so the
// returned value is automatically redacted from subsequent task logs,
// matching Python's airflow.models.variable.Variable.get behaviour.
// Pairs with the "TODO: mask secrets here" hook in
// pkg/worker/runner.go's task log handler — both halves are needed
// before secret masking actually works end-to-end.
```
```go
// go-sdk/sdk/client.go - GetConnection
// TODO: register Connection.Password and sensitive-keyed entries of
// Connection.Extra with a SecretsMasker so they are auto-redacted from
// subsequent task logs, matching Python's
// airflow.models.connection.Connection.get behaviour. Pairs with the
// "TODO: mask secrets here" hook in pkg/worker/runner.go's task log
// handler and the matching TODO on GetVariable above.
```
Coordinator-mode client (lands with #67317):
```go
// go-sdk/pkg/execution/client.go - GetVariable and GetConnection
// (same TODO text, same SecretsMasker hook)
```
Log-handler hook (lands with #67318): the ``// TODO: mask secrets here`` comment in ``go-sdk/pkg/worker/runner.go`` is where the consumer side of the redaction belongs.
### What needs to happen
1. Design and add a ``SecretsMasker`` type to the Go SDK with a register-and-redact API roughly equivalent to ``airflow.utils.log.secrets_masker.SecretsMasker`` (case-insensitive secret-name detection, replace-with-``***`` semantics, per-task scoping).
2. Register secret-named variables in both ``sdk.client.GetVariable`` and ``execution.CoordinatorClient.GetVariable``.
3. Register ``Connection.Password`` and sensitive-keyed entries of ``Connection.Extra`` in both ``sdk.client.GetConnection`` and ``execution.CoordinatorClient.GetConnection``.
4. Wire the masker into the task log handler at the ``// TODO: mask secrets here`` site in ``pkg/worker/runner.go`` so emitted log lines are redacted before they leave the runtime.
5. Remove the four coordinated TODOs above.
### Acceptance criteria
- Looking up a variable named e.g. ``my_password`` via either ``sdk.client`` or ``CoordinatorClient`` causes any subsequent task log line containing the variable's value to be redacted to ``***`` (or the project's chosen sentinel).
- ``Connection.Password`` and sensitive-keyed ``Connection.Extra`` entries produce the same redaction behaviour from both clients.
- Tests cover: (a) registration from each client, (b) redaction in the log handler, (c) negative case where a non-sensitive variable name is not registered.
- All four TODOs cross-referencing this work are removed from the codebase.
### Context
- Originating review: self-review thread on #67317 (coordinator comms + client PR).
- HTTP-side TODOs (already on ``main``): ``go-sdk/sdk/client.go`` ``GetVariable`` and ``GetConnection``.
- Coordinator-side TODOs (land with #67317): ``go-sdk/pkg/execution/client.go`` ``GetVariable`` and ``GetConnection``.
- Log-handler hook (lands with #67318): ``go-sdk/pkg/worker/runner.go``.
- Python parity: ``airflow.utils.log.secrets_masker.SecretsMasker``, ``airflow.models.variable.Variable.get``, ``airflow.models.connection.Connection.get``.
- Related project board: https://github.com/orgs/apache/projects/499/views/1
---
Drafted-by: Claude Code (Opus 4.7) (no human review before posting)
Contributor guide
Assessment
This issue has not been assessed yet.