[RFC] GCP Secret Manager support for configuration secrets

[RFC] GCP Secret Manager support for configuration secrets

This RFC proposes a GCP Secret Manager secrets provider for Data Prepper, so pipeline and server configuration can reference secrets via ${{gcp_secrets:...}} — mirroring the existing AWS Secrets Manager support.

1. Context

Data Prepper resolves secret references in configuration through a pluggable mechanism:

• PluginConfigValueTranslator (data-prepper-api) — getPrefix(), translate(String), translateToPluginConfigVariable(String).
• SecretsSupplier — Object retrieveValue(String secretId, String key) and Object retrieveValue(String secretId).
• VariableExpander (plugin framework) — scans configuration for ${{<prefix>:...}} and dispatches to the translator registered for that prefix.
• Extension registration via @DataPrepperExtensionPlugin + ExtensionPoints.addExtensionProvider(...).

Today the only implementation is AWS (data-prepper-plugins/aws-plugin, prefix aws_secrets). There is no GCP or Azure provider, and no open issue or PR proposing one. Because the mechanism is explicitly pluggable, a GCP provider requires no core changes — only a new module that registers a translator for a new prefix.

2. Motivation

Users running Data Prepper on Google Cloud keep their credentials (OpenSearch passwords, API tokens, database credentials, etc.) in GCP Secret Manager, not AWS Secrets Manager. Today they have no way to reference those secrets from Data Prepper configuration — they must inline plaintext secrets or build external templating, both of which are undesirable for production. Providing a native gcp_secrets provider gives GCP users the same first-class secret-injection experience AWS users already have.

3. Proposal

A new data-prepper-plugins/gcp-plugin module that registers a GCP Secret Manager secrets provider, with parity to the AWS provider: value translation, caching, periodic refresh, and rotation support.

Reference syntax

• ${{gcp_secrets:<secretId>}} — returns the whole secret payload. Use for secrets whose payload is a plain string (the common GCP pattern: one secret holds one value).
• ${{gcp_secrets:<secretId>:<key>}} — parses the payload as JSON and extracts <key>. Use for secrets whose payload is a JSON object.

<secretId> names a secret store defined in data-prepper-config.yaml, which points at the actual GCP secret (the same reference → config-store → cloud-secret indirection AWS uses).

Both forms are supported because a GCP Secret Manager payload is opaque bytes — it may be a plain string or a JSON blob, depending entirely on how the user stored it. The two forms map onto the two existing SecretsSupplier methods (no key → whole payload; key → JSON extraction). Documentation will clearly state which form to use for each secret shape, so users do not apply the :key form to a non-JSON payload.

Versions: v1 always resolves the latest secret version. Explicit version selection is deferred (see Deferred), keeping the reference grammar identical to AWS.

Example

data-prepper-config.yaml:

extension:
  gcp:
    secrets:
      rss-credentials:
        secret_id: rss-credentials
        project_id: my-gcp-project

Pipeline:

rss-pipeline:
  source:
    rss:
      feeds:
        internal:
          url: https://private.example.com/feed.xml
          authentication:
            basic:
              username: "${{gcp_secrets:rss-credentials:username}}"
              password: "${{gcp_secrets:rss-credentials:password}}"

Or, with each credential stored as its own plain-string secret:

              username: "${{gcp_secrets:rss-username}}"
              password: "${{gcp_secrets:rss-password}}"

4. Architecture

The module mirrors the AWS plugin's structure. Shared contracts (SecretsSupplier, PluginConfigValueTranslator) are reused, so VariableExpander and the framework are untouched.

GCP class	AWS analog	Responsibility
GcpSecretPlugin	AwsSecretPlugin	@DataPrepperExtensionPlugin(rootKeyJsonPath = "/gcp/secrets"); apply(ExtensionPoints) wires providers
GcpSecretPluginConfig	AwsSecretPluginConfig	Maps the /gcp/secrets block (map of store-name → configuration)
GcpSecretManagerConfiguration	AwsSecretManagerConfiguration	Per-store: secret_id, project_id, auth options
GcpSecretsSupplier (implements SecretsSupplier)	AwsSecretsSupplier	GCP Secret Manager access; caching; both retrieveValue overloads
GcpSecretsPluginConfigValueTranslator	AwsSecretsPluginConfigValueTranslator	prefix gcp_secrets; optional-:key regex
Extension providers + config publisher	AWS analogs	Registration and refresh publishing
GcpPluginConfigVariable	AwsPluginConfigVariable	Rotation support
Refresh job	SecretsRefreshJob	Periodic refresh on an interval

Data flow

1. At startup the framework loads GcpSecretPlugin; apply() reads /gcp/secrets, builds the supplier, and registers the translator (prefix gcp_secrets) and refresh publisher.
2. During config parse, VariableExpander finds ${{gcp_secrets:...}} and routes to the translator, which calls retrieveValue(secretId) or retrieveValue(secretId, key); the resolved value is substituted in memory before plugin construction.
3. The supplier accesses projects/<project>/secrets/<name>/versions/latest, caches the payload, and (key form) JSON-parses and extracts the field.
4. A refresh job re-fetches on an interval and updates the cache.

Authentication

Uses Application Default Credentials (ADC) by default (the standard GCP chain: GOOGLE_APPLICATION_CREDENTIALS, workload identity, metadata server). Per-store options: project_id (required if not inferable from ADC) and, optionally, an explicit service-account key path or impersonation target — mirroring AWS's credentials-provider chain and STS options.

5. Dependency

This introduces a new dependency: com.google.cloud:google-cloud-secretmanager.

Per the contributing guidance, this RFC explicitly requests maintainer approval for adding this dependency before implementation proceeds. It is the GCP-official SDK for Secret Manager and is required to access secrets.

6. Error handling

• Malformed reference → IllegalArgumentException, as AWS does.
• :key form used on a non-JSON payload → a clear error naming the secret and directing the user to the no-key form.
• Secret/version not found, permission denied, or GCP unavailable → fail fast at startup with an actionable message rather than starting a half-configured pipeline.

7. Testing

• Unit tests for the translator (regex, both reference forms, invalid input) and the supplier (mocked SecretManagerServiceClient: whole payload, JSON key, non-JSON-payload error, caching, refresh), plus config deserialization.
• A module integration test with a mocked GCP client (no real GCP calls in CI) verifying extension registration and end-to-end ${{gcp_secrets:...}} resolution.

8. Deferred / future work

• Explicit version selection (v1 resolves latest).
• An Azure Key Vault provider — a natural follow-up using the same extension pattern; out of scope here.

9. Alternatives considered

• External templating / inlined secrets. Rejected: not production-appropriate and not native to Data Prepper's configuration model.
• Second segment as a version selector instead of a JSON key. Rejected for v1: it would diverge from the AWS reference grammar; version selection is deferred instead.

Start by reading the existing AWS implementation in data-prepper-plugins/aws-plugin and the listed extension points: PluginConfigValueTranslator, SecretsSupplier, VariableExpander, and ExtensionPoints. First obtain maintainer approval for com.google.cloud:google-cloud-secretmanager, then compare the proposed GCP classes and tests with the AWS analogs; done means registration, both reference forms, caching and refresh, configuration handling, and mocked integration coverage.