ozontech / ozontech/file.d

Feature: transform plugin — cover the functionality of the modify plugin

Open
#1,000 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
504
Forks
263
Avg merge
4d 2h
Merged PRs (30d)
5

Description

Is your feature request related to a problem? Please describe.

Pipelines that need modify's substitution filters still have to run both plugins side by side: field
interpolation and conditionals are covered by the transform language, but there is no equivalent of re,
trim, trim_to or cut. That means two syntaxes for conceptually the same work, an extra action per
pipeline, and no way to migrate a modify-based config in one step.

But the gap is not only which filters exist. A modify substitution can only ever produce one value, so
extracting N fields from one line means N independent filter chains re-scanning the same string, each
re-deriving position from scratch:

- type: modify
  shard:   '${log|trim_to("left", "[")|trim_to("right", "]")}'
  time:    '${log|trim_to("left", " ")|trim("left", " ")|trim_to("right", " [")|trim("right", " [")}'
  level:   '${log|trim_to("right", " [")|trim("right", " [")|trim_to("right", " ")|trim("right", " ")|trim_to("right", " ")|trim("right", " ")}'
  message: '${log|trim_to("left", "] ")|trim("left", "] ")}'

Porting the four filters verbatim would let users write exactly this in a new syntax. That is not the point of
the exercise. capture() already collapses the whole block into a single match:

- type: transform
  source: |
    m = capture(.log, r'^(?P<level>\S+)\s+(?P<time>\S+ \S+)\s+\[(?P<shard>[^\]]+)\]\s+(?P<operation>\S+)\s+-\s+(?P<message>.+)$')
    if m != null {
      .level = m.level; .time = m.time; .shard = m.shard; .message = m.message
    }

So the goal of this issue is stated as capability parity, not filter parity:

Every modify config has a transform equivalent, written the way the transform language wants — not
every modify filter has a same-named transform function.

Describe the solution you'd like

The smallest set of stdlib additions that closes the gap: one new named argument on an existing function, and
six new functions, each a small struct on the stdlib.Function shape plus a mustRegister line.

addition covers behaviour
capture(value, pattern, numeric_groups: false) re(...) positional groups existing function; with numeric_groups: true the positional groups also appear under the keys "0", "1", … ("0" is the whole match)
find_all(value, pattern, group: 0, limit: -1) re(...) multiple occurrences array of matched strings; [] when nothing matches; limit: -1 means all
join(array, separator: "") re's separator argument joins an array of strings; not regex-specific
trim(value, cutset) / trim_left / trim_right trim(mode, cutset) strips any of the characters in cutset from both / left / right
slice(value, start, end) cut(mode, count) substring by rune offsets; negative offsets count from the end, like .items[-1]; end defaults to the end of the string
trim_to(mode, cutset) not ported, see below
Migration
modify transform
${message|re("(\w+):.*",-1,[1],",")} capture(.message, r'(\w+):.*', numeric_groups: true)["1"]
${message|re("(re\d+)",2,[1],",")} join(find_all(.message, r're\d+', limit: 2), ",")
${message|re("service=(\S+) exec took (\d+\.?\d*(?:ms|s|m|h))",-1,[2],",")} capture(.message, r'...', numeric_groups: true)["2"]
${message|re("test",1,[1],",",true)} capture / find_all + if, see no-match below
${message|trim("right","\n")} trim_right(.message, "\n")
${message|cut("first",10)} slice(.message, 0, 10)
${message|cut("last",5)} slice(.message, -5)
${log|trim_to("left","[")|trim_to("right","]")} between(.log, "[", "]")
${log|trim_to("left","] ")|trim("left","] ")} after(.log, "] ")
${log|trim_to("left"," ")|trim("left"," ")|trim_to("right"," [")|trim("right"," [")} before(after(.log, " "), " [")
${field} interpolation, _skip_empty covered by paths, + and if

Named capture groups remain the preferred form; numeric_groups exists for patterns that are awkward to
rewrite with names, and for parity with re's groups: [1,2] (m["1"] + "," + m["2"]).

Design decisions

trim_to is not ported. after, before and between already cover it, and they fix two problems it
has. First, trim_to keeps the delimiter it matched
(trim_to_filter.gosrc[idx:] / src[:idx+1]), which is why real
configs always follow it with a trim to delete that delimiter again; every trim_to in the block above is
paired with such an undo. Second, its left mode uses Index while its right mode uses LastIndex, so
trim_to("left","[")|trim_to("right","]") silently returns far more than the first bracketed group on any
line containing more than one ]. between() is unambiguous. If a LastIndex variant is ever needed, it
should be added as after_last / before_last, not as a mode argument.

No mode: string arguments. compiler/validate.go
checks function names and argument shape at startup, but Parameter.AcceptedKinds only checks the value
kind, and only at call time. A typo like mode: "lft" would therefore compile cleanly and fail per
event
, stopping the program partway with some fields already written — a regression against modify, which
rejects a bad mode at config parse. Distinct function names (trim_left, trim_right) are validated at
startup by the registry for free, and read better at the call site.

No empty_on_not_matched. capture already returns null on no match
(capture.go) and find_all returns []; if covers the
empty-string default. Note that modify's re currently returns the unchanged input when the regex misses
(regex_filter.go) — silently writing the whole raw line into .level
which is behaviour worth not reproducing.

Two bugs in the modify filters are fixed rather than inherited. trim's cutset is documented as a
"substring" but is bytes.Trim, i.e. a set of characters: trim("right","ms") strips every trailing m
and s. And cut counts bytes, so it happily splits a UTF-8 rune in half. The new trim* and slice
operate on runes, and the docs say character set.

cfg/substitution implementations are not reused, its test cases are. RegexFilter is stateful and
non-reentrant (it holds a buf mutated on every Apply, wired through setBuffer), and the rest are
[]byte-in/[]byte-out while transform works in core.StringValue. Reuse would buy a concurrency hazard and
conversions in exchange for saving a few lines of strings.Trim. What actually prevents the two plugins from
drifting is a shared test table, so the modify README examples are ported as test cases.

slice follows VRL's slice — same name, same
argument order, negative offsets, optional end. No reason to invent a different convention for the same
operation.

Acceptance criteria
  • numeric_groups added to capture; find_all, join, trim, trim_left, trim_right, slice
    implemented and registered
  • Every example in the modify README has a transform equivalent producing the same event, as a
    table-driven test
  • Unit tests per function: no-match, empty input, out-of-range and negative slice offsets, limit of 0
    and negative, multi-byte input
  • Docs: entries in the functions doc block in transform.go plus a regenerated README
  • A migration section in the docs, led by the "one capture replaces four substitution chains" example

Describe alternatives you've considered

  • Keep using modify alongside transform — the status quo; an extra action per pipeline and two syntaxes
    to learn.
  • Port re / trim / trim_to / cut 1:1, with mode:, groups:, separator: and
    empty_on_not_matched: as named arguments
    — the original draft of this issue. Rejected: it carries
    modify's per-value re-scanning idiom, its runtime-only mode validation, and its byte/cutset bugs into a
    language that does not need any of them.
  • Port the ${field|re(...)} substitution syntax verbatim — eases migration, but bolts a second expression
    language onto one that already has paths, calls and named arguments.
  • One generic filter(value, "re", ...) entry point — fewer registrations, but loses per-function argument
    validation at startup.
  • Adopt more of VRL's string functions
    while we are here
    (replace, split, truncate, downcase, strip_whitespace,
    strip_ansi_escape_codes, the contains / starts_with / match predicate family) — all reasonable
    additions, none of them needed to cover modify. Deliberately left out to keep this change reviewable;
    worth separate issues. In particular the predicates are not required here, because do_if is applied
    generically to every action plugin (pipeline/processor.go), so an existing
    do_if + modify pair migrates to do_if + transform with the condition untouched.

Additional context

Relevant code:

  • plugin/action/modify/modify.go — the plugin being covered
  • cfg/substitution/ — the existing filter implementations, as a behavioural reference
  • plugin/action/transform/stdlib/registry.goFunction interface, Parameter (positional vs. named)
  • plugin/action/transform/stdlib/upcase.go, substring.go — the shape to model the new functions on
  • plugin/action/transform/transform.go — the functions doc block that feeds the README

Out of scope: deprecating or removing the modify plugin, and any change to the transform language itself —
no new operators (including ??), no new syntax. This issue only makes transform capable of replacing
modify.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with plugin/action/transform/stdlib/registry.go and the implementations in upcase.go and substring.go, then inspect capture.go and the existing filters under cfg/substitution/. Add the requested capture option and six registered functions, following the stated rune and no-match behavior. Run unit tests and add table-driven migration coverage, update transform.go's function docs, and regenerate the README.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.