redpanda-data / redpanda-data/benthos

streams API: support per-request `env` overrides for config templating

Open
#481 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
571
Forks
120
Avg merge
2d 1h
Merged PRs (30d)
18

Description

Summary

Add an optional env field to the streams-mode HTTP API's config payloads so a caller can supply per-request environment variable overrides, instead of having to pre-render ${FOO}-style templates themselves or rely on real OS environment variables.

Background

Benthos configs support ${FOO:default|func} templating, resolved by (*config.Reader).ReplaceEnvVariables against an envLookupFunc (default os.LookupEnv). The streams-mode HTTP API already runs every submitted config body through this exact mechanism before parsing (internal/stream/manager/api.go):

  • POST/PUT /streams/{id} (HandleStreamCRUD)
  • POST /resources/{type}/{id} (HandleResourceCRUD)

Today the only source of template values is the real process environment, so posting "the same stream config with different parameters" means the caller must pre-render the template themselves, or run separate processes with different real env vars. There's no way to parameterize a stream config per-request through the API itself.

Proposal

Support an optional, top-level env field alongside the normal stream config fields in the request body:

{
  "env": { "VAR1": "val1", "VAR2": "val2" },
  "input": { ... },
  "output": { ... }
}

env values fill in ${VAR1}-style placeholders anywhere else in the document, taking precedence over a same-named real OS environment variable. This introduces no new templating syntax — it only changes where the existing envLookupFunc seam gets its answers from, reusing config.OptUseEnvLookupFunc (already used elsewhere, e.g. public/service/resource_builder.go, config_querier.go).

Scope
  • POST/PUT /streams/{id} (HandleStreamCRUD) — primary target.
  • POST /resources/{type}/{id} (HandleResourceCRUD) — included, identical seam.
  • Bulk POST /streams (HandleStreamsCRUD) is out of scope for this issue. It currently has no env-var substitution at all (a pre-existing, unrelated gap), so adding overrides there means building that support for the first time rather than reusing it.
  • env values must be strings, matching the existing envLookupFunc contract (func(context.Context, string) (string, bool), backed by os.LookupEnv, which only ever returns strings). A non-string value should be a request error (400), not silently coerced.
Why env needs to be stripped before linting

docs.FieldSpecs.LintYAML (internal/docs/format_yaml.go) raises an error-level LintUnknown for any top-level field not in the component's spec, and both HandleStreamCRUD and HandleResourceCRUD treat any non-empty lint list as a hard 400 (unless ?chilled=true). So env can't just be left in the document for the existing lint/parse calls to ignore — it needs to be extracted and removed from the raw config before linting and parsing, and used only to build the lookup func.

Suggested implementation sketch

All in internal/stream/manager/api.go (package manager):

  1. A helper to extract-and-strip a top-level env field from the raw config bytes (decode → gabs.Wrap → mutate → re-encode, following the same round-trip pattern already used by patchConfig for merging a PATCH body into an existing config).
  2. A helper building a precedence-aware lookup func: check the request-supplied overrides map first, fall back to os.LookupEnv.
  3. Wire both into HandleStreamCRUD's config-reading path: extract/strip env from confBytes, then pass a config.NewReader("", nil, config.OptUseEnvLookupFunc(...)) (only when overrides are non-empty) into the existing ReplaceEnvVariables call. Everything downstream (lint, ParsedConfigFromAny, stream.FromParsed, chilled/ErrMissingEnvVars handling) stays untouched.
  4. Apply the same pattern to HandleResourceCRUD's config-reading path.
  5. Update the two RegisterEndpoint description strings for /streams/{id} and /resources/{type}/{id} to document the new optional env field.

Acceptance criteria

  • Posting a stream/resource config with an env map resolves ${VAR} placeholders in the rest of the document using those values.
  • A request env value takes precedence over a same-named real OS environment variable.
  • Omitting env behaves exactly as today (falls back to OS env vars only) — no behavior change for existing callers.
  • A missing variable (absent from both request env and OS env) still produces today's ErrMissingEnvVars/chilled behavior, unchanged.
  • A non-string env value is rejected with a 400, not coerced.
  • Test coverage in internal/stream/manager/api_test.go for: override applied, override precedence over OS env, fallback to OS env when no override given, missing-var error unchanged, non-string value rejected, and a no-env regression check — for both /streams/{id} and /resources/{type}/{id}.

Out of scope

  • Bulk POST /streams (HandleStreamsCRUD) — no existing env substitution there to extend.
  • Any new templating syntax beyond the existing ${FOO} mechanism.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in internal/stream/manager/api.go by reading HandleStreamCRUD, HandleResourceCRUD, patchConfig, and the existing ReplaceEnvVariables calls. Then inspect internal/stream/manager/api_test.go and the endpoint descriptions. Add coverage for overrides, precedence, OS fallback, missing variables, invalid values, and no-env behavior for both endpoints; done means the acceptance criteria pass without changing bulk POST /streams.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.