ozontech / ozontech/file.d

Feature: transform plugin — translate field values via a configured mapping

Open
#999 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
504
Forks
263
Avg merge
4d 2h
Merged PRs (30d)
5

Description

Is your feature request related to a problem? Please describe.
There's currently no built-in way to translate a field's raw value into a more human-readable one using a static lookup table. For example, when processing Kafka logs, the kafka_request_api_key field contains numeric API key codes ("0", "1", "2", ...) instead of their human-readable names (produce, fetch, list_offsets, ...). Today this requires an if/else chain in the transform program source, which doesn't scale to the ~70 entries of the Kafka API key table.

Describe the solution you'd like
Add a lookup(value, table, default:) function to the transform stdlib. The table is an ordinary object value, so no new config parameter is needed:

actions:
  - type: transform
    source: |
      api_key = {"0": "produce", "1": "fetch", "2": "list_offsets", "3": "metadata"}
      .kafka_request_api_key = lookup(.kafka_request_api_key, api_key)

Behavior notes:

  • value — the value to translate; reading a missing field yields null and passes through unchanged.
  • table — an object mapping raw value → replacement value. Objects only.
  • Keys are matched by their canonical string form, so the number 0 and the string "0" address the same entry — JSON writes enum codes both ways.
  • A value that isn't a key of the table is returned unchanged, rather than erroring or emptying the field. default: overrides that: lookup(.status, {"500": "crit"}, default: "ok").
  • Composite values (arrays, objects) are rejected as keys.

For this to be usable with large tables, a table literal must not be rebuilt on every event — ObjectExpr.Eval currently allocates a fresh map per evaluation. Constant sub-expressions should be folded into a new ConstExpr AST node inside the existing validation walk (ValidateCalls), which already prepares nodes in place: it compiles regex literals and parses timestamp literals. No separate compiler pass.

  • Fold in two positions: function call arguments and assignment right-hand sides, so both lookup(.x, {...}) and the table = {...} idiom are covered.
  • An expression that fails to evaluate must never be folded — it stays in the AST and keeps failing per event, so folding can't turn a runtime error into a startup failure.
  • Folded values are shared across events and processor goroutines. That's safe only because values are immutable (index and member assignment copy the container), and it must be covered by a test.

Goal: the per-event cost of a lookup doesn't grow with the size of the table.

Describe alternatives you've considered

  • A mapping: section in the plugin config plus map(value, "name") (the shape originally proposed here) — rejected: it adds a config parameter serving exactly one function, and every future function needing startup-prepared data would want its own. Resolving a table by name at runtime also needs per-instance state, but the stdlib registry is a process-global of stateless functions, so it would mean threading a per-instance registry through Start, ValidateCalls and NewContext. And since validating table names at startup requires a literal argument anyway, the runtime name lookup buys nothing. Finally, a per-action mapping: block is no more reusable across actions than an object literal is.
  • Inline if/else chains in the program source — doesn't scale and is hard to maintain for enums with many values.
  • An object literal without constant folding — correct, but rebuilds the table on every event.
  • Loading the table from an external file — out of scope for now; see below.
  • Doing the mapping upstream — not always possible when the source of the field (e.g. a Kafka producer) can't be changed.

Additional context
Example use case: Kafka kafka_request_api_key numeric codes → human-readable API names (produce, fetch, offsets, etc.), to make dashboards and log search more readable without needing a lookup table on the query side.

Name the function lookup, not map — in an expression language map reads as map/filter/reduce.

Out of scope:

  • loading tables from an external file — additive later as a foldable from_file(...) in the same argument position, if large tables in YAML become a real complaint;
  • a Const flag on Parameter to reject a non-constant table at startup; a non-constant table should simply work and not be folded.

Definition of done:

  • unit tests for lookup (hits, miss behavior, default:, numeric/string key equivalence, argument validation);
  • unit tests for folding, including the don't-fold-on-error rule and the shared-value immutability guarantee;
  • an end-to-end test in transform_test.go;
  • lookup documented in the plugin's functions section, with README.md regenerated via make gen-doc.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the transform stdlib, ObjectExpr.Eval, and the existing ValidateCalls walk to understand function arguments, assignment right-hand sides, and literal preparation. Add unit coverage for lookup and constant folding, including error handling and shared-value immutability, then run the end-to-end cases in transform_test.go. Finish by documenting the function and regenerating README.md with make gen-doc.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.