Feature: transform plugin — translate field values via a configured mapping
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 504
- Forks
- 263
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 5
Description
Is your feature request related to a problem? Please describe.
There's currently no built-in way to translate a field's raw value into a more human-readable one using a static lookup table. For example, when processing Kafka logs, the kafka_request_api_key field contains numeric API key codes ("0", "1", "2", ...) instead of their human-readable names (produce, fetch, list_offsets, ...). Today this requires an if/else chain in the transform program source, which doesn't scale to the ~70 entries of the Kafka API key table.
Describe the solution you'd like
Add a lookup(value, table, default:) function to the transform stdlib. The table is an ordinary object value, so no new config parameter is needed:
actions:
- type: transform
source: |
api_key = {"0": "produce", "1": "fetch", "2": "list_offsets", "3": "metadata"}
.kafka_request_api_key = lookup(.kafka_request_api_key, api_key)
Behavior notes:
value— the value to translate; reading a missing field yieldsnulland passes through unchanged.table— an object mapping raw value → replacement value. Objects only.- Keys are matched by their canonical string form, so the number
0and the string"0"address the same entry — JSON writes enum codes both ways. - A value that isn't a key of the table is returned unchanged, rather than erroring or emptying the field.
default:overrides that:lookup(.status, {"500": "crit"}, default: "ok"). - Composite values (arrays, objects) are rejected as keys.
For this to be usable with large tables, a table literal must not be rebuilt on every event — ObjectExpr.Eval currently allocates a fresh map per evaluation. Constant sub-expressions should be folded into a new ConstExpr AST node inside the existing validation walk (ValidateCalls), which already prepares nodes in place: it compiles regex literals and parses timestamp literals. No separate compiler pass.
- Fold in two positions: function call arguments and assignment right-hand sides, so both
lookup(.x, {...})and thetable = {...}idiom are covered. - An expression that fails to evaluate must never be folded — it stays in the AST and keeps failing per event, so folding can't turn a runtime error into a startup failure.
- Folded values are shared across events and processor goroutines. That's safe only because values are immutable (index and member assignment copy the container), and it must be covered by a test.
Goal: the per-event cost of a lookup doesn't grow with the size of the table.
Describe alternatives you've considered
- A
mapping:section in the plugin config plusmap(value, "name")(the shape originally proposed here) — rejected: it adds a config parameter serving exactly one function, and every future function needing startup-prepared data would want its own. Resolving a table by name at runtime also needs per-instance state, but the stdlib registry is a process-global of stateless functions, so it would mean threading a per-instance registry throughStart,ValidateCallsandNewContext. And since validating table names at startup requires a literal argument anyway, the runtime name lookup buys nothing. Finally, a per-actionmapping:block is no more reusable across actions than an object literal is. - Inline if/else chains in the program source — doesn't scale and is hard to maintain for enums with many values.
- An object literal without constant folding — correct, but rebuilds the table on every event.
- Loading the table from an external file — out of scope for now; see below.
- Doing the mapping upstream — not always possible when the source of the field (e.g. a Kafka producer) can't be changed.
Additional context
Example use case: Kafka kafka_request_api_key numeric codes → human-readable API names (produce, fetch, offsets, etc.), to make dashboards and log search more readable without needing a lookup table on the query side.
Name the function lookup, not map — in an expression language map reads as map/filter/reduce.
Out of scope:
- loading tables from an external file — additive later as a foldable
from_file(...)in the same argument position, if large tables in YAML become a real complaint; - a
Constflag onParameterto reject a non-constant table at startup; a non-constant table should simply work and not be folded.
Definition of done:
- unit tests for
lookup(hits, miss behavior,default:, numeric/string key equivalence, argument validation); - unit tests for folding, including the don't-fold-on-error rule and the shared-value immutability guarantee;
- an end-to-end test in
transform_test.go; lookupdocumented in the plugin's functions section, withREADME.mdregenerated viamake gen-doc.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the transform stdlib, ObjectExpr.Eval, and the existing ValidateCalls walk to understand function arguments, assignment right-hand sides, and literal preparation. Add unit coverage for lookup and constant folding, including error handling and shared-value immutability, then run the end-to-end cases in transform_test.go. Finish by documenting the function and regenerating README.md with make gen-doc.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100