Feature: transform plugin — cover the functionality of the modify plugin
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 504
- Forks
- 263
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 5
Description
Is your feature request related to a problem? Please describe.
Pipelines that need modify's substitution filters still have to run both plugins side by side: field
interpolation and conditionals are covered by the transform language, but there is no equivalent of re,
trim, trim_to or cut. That means two syntaxes for conceptually the same work, an extra action per
pipeline, and no way to migrate a modify-based config in one step.
But the gap is not only which filters exist. A modify substitution can only ever produce one value, so
extracting N fields from one line means N independent filter chains re-scanning the same string, each
re-deriving position from scratch:
- type: modify
shard: '${log|trim_to("left", "[")|trim_to("right", "]")}'
time: '${log|trim_to("left", " ")|trim("left", " ")|trim_to("right", " [")|trim("right", " [")}'
level: '${log|trim_to("right", " [")|trim("right", " [")|trim_to("right", " ")|trim("right", " ")|trim_to("right", " ")|trim("right", " ")}'
message: '${log|trim_to("left", "] ")|trim("left", "] ")}'
Porting the four filters verbatim would let users write exactly this in a new syntax. That is not the point of
the exercise. capture() already collapses the whole block into a single match:
- type: transform
source: |
m = capture(.log, r'^(?P<level>\S+)\s+(?P<time>\S+ \S+)\s+\[(?P<shard>[^\]]+)\]\s+(?P<operation>\S+)\s+-\s+(?P<message>.+)$')
if m != null {
.level = m.level; .time = m.time; .shard = m.shard; .message = m.message
}
So the goal of this issue is stated as capability parity, not filter parity:
Every
modifyconfig has atransformequivalent, written the way the transform language wants — not
everymodifyfilter has a same-namedtransformfunction.
Describe the solution you'd like
The smallest set of stdlib additions that closes the gap: one new named argument on an existing function, and
six new functions, each a small struct on the stdlib.Function shape plus a mustRegister line.
| addition | covers | behaviour |
|---|---|---|
capture(value, pattern, numeric_groups: false) |
re(...) positional groups |
existing function; with numeric_groups: true the positional groups also appear under the keys "0", "1", … ("0" is the whole match) |
find_all(value, pattern, group: 0, limit: -1) |
re(...) multiple occurrences |
array of matched strings; [] when nothing matches; limit: -1 means all |
join(array, separator: "") |
re's separator argument |
joins an array of strings; not regex-specific |
trim(value, cutset) / trim_left / trim_right |
trim(mode, cutset) |
strips any of the characters in cutset from both / left / right |
slice(value, start, end) |
cut(mode, count) |
substring by rune offsets; negative offsets count from the end, like .items[-1]; end defaults to the end of the string |
| — | trim_to(mode, cutset) |
not ported, see below |
Migration
| modify | transform |
|---|---|
${message|re("(\w+):.*",-1,[1],",")} |
capture(.message, r'(\w+):.*', numeric_groups: true)["1"] |
${message|re("(re\d+)",2,[1],",")} |
join(find_all(.message, r're\d+', limit: 2), ",") |
${message|re("service=(\S+) exec took (\d+\.?\d*(?:ms|s|m|h))",-1,[2],",")} |
capture(.message, r'...', numeric_groups: true)["2"] |
${message|re("test",1,[1],",",true)} |
capture / find_all + if, see no-match below |
${message|trim("right","\n")} |
trim_right(.message, "\n") |
${message|cut("first",10)} |
slice(.message, 0, 10) |
${message|cut("last",5)} |
slice(.message, -5) |
${log|trim_to("left","[")|trim_to("right","]")} |
between(.log, "[", "]") |
${log|trim_to("left","] ")|trim("left","] ")} |
after(.log, "] ") |
${log|trim_to("left"," ")|trim("left"," ")|trim_to("right"," [")|trim("right"," [")} |
before(after(.log, " "), " [") |
${field} interpolation, _skip_empty |
covered by paths, + and if |
Named capture groups remain the preferred form; numeric_groups exists for patterns that are awkward to
rewrite with names, and for parity with re's groups: [1,2] (m["1"] + "," + m["2"]).
Design decisions
trim_to is not ported. after, before and between already cover it, and they fix two problems it
has. First, trim_to keeps the delimiter it matched
(trim_to_filter.go — src[idx:] / src[:idx+1]), which is why real
configs always follow it with a trim to delete that delimiter again; every trim_to in the block above is
paired with such an undo. Second, its left mode uses Index while its right mode uses LastIndex, so
trim_to("left","[")|trim_to("right","]") silently returns far more than the first bracketed group on any
line containing more than one ]. between() is unambiguous. If a LastIndex variant is ever needed, it
should be added as after_last / before_last, not as a mode argument.
No mode: string arguments. compiler/validate.go
checks function names and argument shape at startup, but Parameter.AcceptedKinds only checks the value
kind, and only at call time. A typo like mode: "lft" would therefore compile cleanly and fail per
event, stopping the program partway with some fields already written — a regression against modify, which
rejects a bad mode at config parse. Distinct function names (trim_left, trim_right) are validated at
startup by the registry for free, and read better at the call site.
No empty_on_not_matched. capture already returns null on no match
(capture.go) and find_all returns []; if covers the
empty-string default. Note that modify's re currently returns the unchanged input when the regex misses
(regex_filter.go) — silently writing the whole raw line into .level —
which is behaviour worth not reproducing.
Two bugs in the modify filters are fixed rather than inherited. trim's cutset is documented as a
"substring" but is bytes.Trim, i.e. a set of characters: trim("right","ms") strips every trailing m
and s. And cut counts bytes, so it happily splits a UTF-8 rune in half. The new trim* and slice
operate on runes, and the docs say character set.
cfg/substitution implementations are not reused, its test cases are. RegexFilter is stateful and
non-reentrant (it holds a buf mutated on every Apply, wired through setBuffer), and the rest are
[]byte-in/[]byte-out while transform works in core.StringValue. Reuse would buy a concurrency hazard and
conversions in exchange for saving a few lines of strings.Trim. What actually prevents the two plugins from
drifting is a shared test table, so the modify README examples are ported as test cases.
slice follows VRL's slice — same name, same
argument order, negative offsets, optional end. No reason to invent a different convention for the same
operation.
Acceptance criteria
-
numeric_groupsadded tocapture;find_all,join,trim,trim_left,trim_right,slice
implemented and registered - Every example in the modify README has a transform equivalent producing the same event, as a
table-driven test - Unit tests per function: no-match, empty input, out-of-range and negative
sliceoffsets,limitof 0
and negative, multi-byte input - Docs: entries in the
functionsdoc block intransform.goplus a regenerated README - A migration section in the docs, led by the "one
capturereplaces four substitution chains" example
Describe alternatives you've considered
- Keep using
modifyalongsidetransform— the status quo; an extra action per pipeline and two syntaxes
to learn. - Port
re/trim/trim_to/cut1:1, withmode:,groups:,separator:and
empty_on_not_matched:as named arguments — the original draft of this issue. Rejected: it carries
modify's per-value re-scanning idiom, its runtime-only mode validation, and its byte/cutset bugs into a
language that does not need any of them. - Port the
${field|re(...)}substitution syntax verbatim — eases migration, but bolts a second expression
language onto one that already has paths, calls and named arguments. - One generic
filter(value, "re", ...)entry point — fewer registrations, but loses per-function argument
validation at startup. - Adopt more of VRL's string functions
while we are here (replace,split,truncate,downcase,strip_whitespace,
strip_ansi_escape_codes, thecontains/starts_with/matchpredicate family) — all reasonable
additions, none of them needed to covermodify. Deliberately left out to keep this change reviewable;
worth separate issues. In particular the predicates are not required here, becausedo_ifis applied
generically to every action plugin (pipeline/processor.go), so an existing
do_if+modifypair migrates todo_if+transformwith the condition untouched.
Additional context
Relevant code:
plugin/action/modify/modify.go— the plugin being coveredcfg/substitution/— the existing filter implementations, as a behavioural referenceplugin/action/transform/stdlib/registry.go—Functioninterface,Parameter(positional vs. named)plugin/action/transform/stdlib/upcase.go,substring.go— the shape to model the new functions onplugin/action/transform/transform.go— thefunctionsdoc block that feeds the README
Out of scope: deprecating or removing the modify plugin, and any change to the transform language itself —
no new operators (including ??), no new syntax. This issue only makes transform capable of replacing
modify.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with plugin/action/transform/stdlib/registry.go and the implementations in upcase.go and substring.go, then inspect capture.go and the existing filters under cfg/substitution/. Add the requested capture option and six registered functions, following the stated rune and no-match behavior. Run unit tests and add table-driven migration coverage, update transform.go's function docs, and regenerate the README.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100