Regex helpers recompile their pattern on every call

Regex helpers recompile their pattern on every call

Summary

regex, regex_all, regex_any and replace_regex call regexp.Compile each
time they are invoked:

// dsl.go:774
MustAddFunction(NewWithPositionalArgs("regex", 2, true, func(args ...interface{}) (interface{}, error) {
	compiled, err := regexp.Compile(toString(args[0]))
	...

In the usual usage the pattern is a constant in the expression while the subject
changes on each evaluation, so the same pattern is compiled over and over.
Compilation is the expensive part of these helpers, and it depends only on the
first argument.

Why the existing result cache does not cover it

dslFunction.Exec caches results for functions registered as cacheable, which
regex is. The key is a hash of the function name and every argument:

// func.go:59
functionHash := d.hash(args...)
if result, err := resultCache.Get(functionHash); err == nil {
    return result, nil
}
result, err := d.ExpressionFunction(args...)   // compiles the pattern

Whenever the subject differs between calls — the common case for an expression
evaluated against changing input — the key differs too, the cache misses, and
the pattern is compiled again. The caching is at call granularity; the
reusable work is at pattern granularity.

Proposed change

Keep compiled patterns in a cache keyed on the pattern, bounded by the existing
gcache and DefaultCacheSize so patterns built at runtime cannot grow it
without limit. *regexp.Regexp is safe for concurrent use, so one compilation
can serve every caller.

compiledRegexCache = gcache.New[string, *regexp.Regexp](DefaultCacheSize).Build()

func compileRegex(pattern string) (*regexp.Regexp, error) {
	if compiled, err := compiledRegexCache.GetIFPresent(pattern); err == nil {
		return compiled, nil
	}

	compiled, err := regexp.Compile(pattern)
	if err != nil {
		return nil, err
	}
	_ = compiledRegexCache.Set(pattern, compiled)

	return compiled, nil
}

Applied at dsl.go:283, :774, :788 and :810. go test ./... passes
unchanged.

Benchmark

Calling the registered regex function with a constant pattern and a unique
~19 KB subject per iteration, so the result cache cannot hit:

variant	ns/op	B/op	allocs/op
v0.8.20	392,206	28,182	60
with compiled-pattern cache	387,793	19,219	7

About 9 KB and 53 allocations of library overhead removed per call; the ~19 KB
that remains is the benchmark building its own subject. Latency is unchanged —
the saving is allocation and the GC pressure that follows from it, not wall
clock.

Impact on a consumer

Found while profiling Nuclei,
which evaluates these helpers once per HTTP response. Over 14 hours in one
long-running process, alloc_space totalled 943 GB, of which this single line
accounted for 259 GB — 27.5% of everything the process allocated:

ROUTINE ======================== github.com/projectdiscovery/dsl.init.0.func50
         0   259.37GB (flat, cum) 27.49% of Total
         .          .    773:	MustAddFunction(NewWithPositionalArgs("regex", 2, true, func(args ...interface{}) (interface{}, error) {
         .   259.36GB    774:		compiled, err := regexp.Compile(toString(args[0]))

Any consumer evaluating a constant-pattern expression against many inputs will
see the same shape, in proportion to how often it evaluates.

A separate observation, not part of this change

For a function whose arguments include a large value, building the result-cache
key costs more than the call it is meant to save. hash copies each string
argument and runs FNV over all of it:

// func.go:81
hasher.Write([]byte(v))

Measured alone that is roughly 23 µs and 24 KB for a 35 KB argument — more than
compiling a pattern — and when the argument is unique the entry it writes can
never be read, while evicting entries that could be.

Marking regex non-cacheable does improve the benchmark further (369,618 ns/op,
19,136 B/op, 4 allocs/op), but I am not proposing it: where the same expression
is evaluated against the same input more than once the result cache presumably
does hit, and I have not measured that case. Raising it in case excluding large
arguments from the key is worth considering.

Start with the regex helper registrations in dsl.go around lines 283, 774, 788, and 810, then read func.go around line 59 to understand the existing result cache and inspect the gcache usage. Run go test ./... and the described benchmark to verify compiled patterns are reused, cache growth is bounded, and existing behavior remains unchanged.