ConduitIO / ConduitIO/conduit

Redact only sensitive connector config values in logs (keyed on real sensitivity metadata)

Open
#2,566 1 comment 0 reactions 0 assignees View on GitHub
roadmap security
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

## Summary

`pkg/foundation/log.RedactAll` (added to fix secrets leaking into logs via
raw `config.Config` dumps - see the PR this issue is filed from) redacts
**every** value in a connector's config when logging it, because
`conduit-commons` `config.Parameter` has no field marking a parameter as
sensitive. Keys stay visible; values don't, sensitive or not.

That's a deliberate interim tradeoff (see the doc comment on
`RedactedConfig` in `pkg/foundation/log/redact.go`), not the intended
end state. This issue tracks doing it properly: redact only parameters
that are actually secret, using real per-parameter metadata instead of a
name heuristic.

## Why this matters

Right now a connector's `topic`, `format`, `batch.size`, etc. all get
printed as `***` at debug level right alongside `sasl.password` and a
DB `url` with an embedded credential. That's safe but not useful - once
this lands, debug logs stay actionable (`topic=orders`) while still
never printing a password.

## Proposed work

1. **conduit-commons**: add a sensitivity flag to `config.Parameter`
(e.g. `ParameterTypeSecret` or `Parameter.Sensitive bool`).
2. **conduit-connector-protocol**: thread the flag through the specifier
proto so it survives the wire (`pconnector.Specification` /
parameter specs already sent by `Specify()`).
3. **conduit-connector-sdk**: expose it via the `paramgen` struct tag
(e.g. `validate:"secret"` or a dedicated tag) so connector authors
opt in without hand-writing specs.
4. **Built-in connectors**: mark the actually-secret parameters (DB
passwords, SASL credentials, API keys, etc.) across the connectors
in this repo and in `conduit-connector-*`.
5. **pkg/foundation/log**: flip the call sites currently using
`log.RedactAll(cfg)` to pass the real `config.Parameters` instead of
nil, so `RedactedConfig.MarshalZerologObject` / `isSensitive` can
consult `Parameter.Sensitive` before falling back to the name
heuristic. `isSensitive`/`isSensitiveParameter` already exist and are
unit-tested ahead of this - only the plumbing to get real `Params` to
each call site is missing.
6. Update the name-heuristic fallback list in `isSensitive` based on
what step 4 turns up (it may be redundant once real metadata exists
everywhere, but should stay as defense-in-depth for connectors that
haven't been updated yet).

## Acceptance criteria

- A connector's non-secret parameters appear in plaintext in debug logs;
parameters marked sensitive (or matching the name heuristic, for
connectors not yet updated) are redacted.
- Existing secret-fixture tests in `pkg/foundation/log`,
`pkg/plugin/connector/builtin`, and `pkg/connector` continue to pass
unmodified (they assert sentinels never leak - that must remain true
under the keyed version too) plus new tests asserting non-secret
values are no longer redacted once `Params` is populated.
- Migration/versioning note for the protocol change per CLAUDE.md's
connector-protocol rules (breaking-change territory).

## Related

- Execution plan: `docs/design-documents/20260704-phase-1-execution-plan.md`
§1.1 edge case: "Secrets in errors/logs → redaction pass keyed on
connector param sensitivity; test with a secret fixture."
- Introduced by the secrets-redaction-in-logs PR (interim
redact-everything behavior): see `pkg/foundation/log/redact.go`.

Contributor guide

Open the contributing guide

Research direction

Start with pkg/foundation/log/redact.go and its existing sensitivity tests, then trace the connector specification and parameter paths through conduit-commons, conduit-connector-protocol, and conduit-connector-sdk. Read CLAUDE.md's connector-protocol rules and the execution-plan reference before changing the wire format. Done means real sensitivity metadata reaches log call sites, secret fixtures still pass, non-secret values remain visible, and the required migration note and tests are present.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, data-engineering, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.