redpanda-data / redpanda-data/benthos
streams API: support per-request `env` overrides for config templating
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 571
- Forks
- 120
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 18
Description
Summary
Add an optional env field to the streams-mode HTTP API's config payloads so a caller can supply per-request environment variable overrides, instead of having to pre-render ${FOO}-style templates themselves or rely on real OS environment variables.
Background
Benthos configs support ${FOO:default|func} templating, resolved by (*config.Reader).ReplaceEnvVariables against an envLookupFunc (default os.LookupEnv). The streams-mode HTTP API already runs every submitted config body through this exact mechanism before parsing (internal/stream/manager/api.go):
POST/PUT /streams/{id}(HandleStreamCRUD)POST /resources/{type}/{id}(HandleResourceCRUD)
Today the only source of template values is the real process environment, so posting "the same stream config with different parameters" means the caller must pre-render the template themselves, or run separate processes with different real env vars. There's no way to parameterize a stream config per-request through the API itself.
Proposal
Support an optional, top-level env field alongside the normal stream config fields in the request body:
{
"env": { "VAR1": "val1", "VAR2": "val2" },
"input": { ... },
"output": { ... }
}
env values fill in ${VAR1}-style placeholders anywhere else in the document, taking precedence over a same-named real OS environment variable. This introduces no new templating syntax — it only changes where the existing envLookupFunc seam gets its answers from, reusing config.OptUseEnvLookupFunc (already used elsewhere, e.g. public/service/resource_builder.go, config_querier.go).
Scope
POST/PUT /streams/{id}(HandleStreamCRUD) — primary target.POST /resources/{type}/{id}(HandleResourceCRUD) — included, identical seam.- Bulk
POST /streams(HandleStreamsCRUD) is out of scope for this issue. It currently has no env-var substitution at all (a pre-existing, unrelated gap), so adding overrides there means building that support for the first time rather than reusing it. envvalues must be strings, matching the existingenvLookupFunccontract (func(context.Context, string) (string, bool), backed byos.LookupEnv, which only ever returns strings). A non-string value should be a request error (400), not silently coerced.
Why env needs to be stripped before linting
docs.FieldSpecs.LintYAML (internal/docs/format_yaml.go) raises an error-level LintUnknown for any top-level field not in the component's spec, and both HandleStreamCRUD and HandleResourceCRUD treat any non-empty lint list as a hard 400 (unless ?chilled=true). So env can't just be left in the document for the existing lint/parse calls to ignore — it needs to be extracted and removed from the raw config before linting and parsing, and used only to build the lookup func.
Suggested implementation sketch
All in internal/stream/manager/api.go (package manager):
- A helper to extract-and-strip a top-level
envfield from the raw config bytes (decode →gabs.Wrap→ mutate → re-encode, following the same round-trip pattern already used bypatchConfigfor merging a PATCH body into an existing config). - A helper building a precedence-aware lookup func: check the request-supplied overrides map first, fall back to
os.LookupEnv. - Wire both into
HandleStreamCRUD's config-reading path: extract/stripenvfromconfBytes, then pass aconfig.NewReader("", nil, config.OptUseEnvLookupFunc(...))(only when overrides are non-empty) into the existingReplaceEnvVariablescall. Everything downstream (lint,ParsedConfigFromAny,stream.FromParsed,chilled/ErrMissingEnvVarshandling) stays untouched. - Apply the same pattern to
HandleResourceCRUD's config-reading path. - Update the two
RegisterEndpointdescription strings for/streams/{id}and/resources/{type}/{id}to document the new optionalenvfield.
Acceptance criteria
- Posting a stream/resource config with an
envmap resolves${VAR}placeholders in the rest of the document using those values. - A request
envvalue takes precedence over a same-named real OS environment variable. - Omitting
envbehaves exactly as today (falls back to OS env vars only) — no behavior change for existing callers. - A missing variable (absent from both request
envand OS env) still produces today'sErrMissingEnvVars/chilledbehavior, unchanged. - A non-string
envvalue is rejected with a 400, not coerced. - Test coverage in
internal/stream/manager/api_test.gofor: override applied, override precedence over OS env, fallback to OS env when no override given, missing-var error unchanged, non-string value rejected, and a no-envregression check — for both/streams/{id}and/resources/{type}/{id}.
Out of scope
- Bulk
POST /streams(HandleStreamsCRUD) — no existing env substitution there to extend. - Any new templating syntax beyond the existing
${FOO}mechanism.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in internal/stream/manager/api.go by reading HandleStreamCRUD, HandleResourceCRUD, patchConfig, and the existing ReplaceEnvVariables calls. Then inspect internal/stream/manager/api_test.go and the endpoint descriptions. Add coverage for overrides, precedence, OS fallback, missing variables, invalid values, and no-env behavior for both endpoints; done means the acceptance criteria pass without changing bulk POST /streams.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100