[performance-profiler] Avoid duplicate filename map work in filestream GetFiles non-overlap scans
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
## Hot Path
`filebeat/input/filestream/fscanner.go:309-323` and `filebeat/input/filestream/fscanner.go:345-352` in `newScanState`/`process`.
`process` currently checks `uniqueFiles` for every matched file. For non-overlapping path sets, that duplicate-filter map is unnecessary work and allocation pressure.
## Profiling Data
**Before:**
```text
Command:
BENCH_TREE_FILES=20000 go test -run '^$' -bench '^(BenchmarkGetFilesSelective|BenchmarkGetFilesExcludeMost|BenchmarkGetFilesManyPatterns|BenchmarkGetFilesLiteralMidComponent|BenchmarkGetFilesMixed)$' -benchmem -count=3 ./filebeat/input/filestream
BenchmarkGetFilesExcludeMost
105087054 ns/op 15961360 B/op 232235 allocs/op
123219598 ns/op 15978988 B/op 232249 allocs/op
125129221 ns/op 15974268 B/op 232245 allocs/op
```
## Proposed Change
Allocate and use `uniqueFiles` only when overlap is possible (`s.pathsCanOverlap`), and skip duplicate-file map checks otherwise.
```diff
--- a/filebeat/input/filestream/fscanner.go
+++ b/filebeat/input/filestream/fscanner.go
@@
func (s *fileScanner) newScanState(opts loginp.FileScanOptions) *scanState {
+ var uniqueFiles map[string]struct{}
+ if s.pathsCanOverlap {
+ uniqueFiles = make(map[string]struct{}, s.lastCount)
+ }
@@
- uniqueFiles: make(map[string]struct{}, s.lastCount),
+ uniqueFiles: uniqueFiles,
@@
- if _, knownFile := st.uniqueFiles[filename]; knownFile {
- st.metrics.FilesNoIngestTarget++
- return
- }
- st.uniqueFiles[filename] = struct{}{}
+ if st.uniqueFiles != nil {
+ if _, knownFile := st.uniqueFiles[filename]; knownFile {
+ st.metrics.FilesNoIngestTarget++
+ return
+ }
+ st.uniqueFiles[filename] = struct{}{}
+ }
```
## Results
**After:**
```text
Command:
BENCH_TREE_FILES=20000 go test -run '^$' -bench '^(BenchmarkGetFilesSelective|BenchmarkGetFilesExcludeMost|BenchmarkGetFilesManyPatterns|BenchmarkGetFilesLiteralMidComponent|BenchmarkGetFilesMixed)$' -benchmem -count=3 ./filebeat/input/filestream
BenchmarkGetFilesExcludeMost
90112158 ns/op 13913980 B/op 232062 allocs/op
91235473 ns/op 14205879 B/op 232080 allocs/op
98174854 ns/op 14201968 B/op 232076 allocs/op
```
**Improvement:** mean `ns/op` improved **20.91%**; mean `B/op` improved **11.67%** for `BenchmarkGetFilesExcludeMost`.
## Verification
- `go test ./filebeat/input/filestream -run '^(TestGetFiles_GrowingRawSuppression|TestGetFiles_DuplicateFingerprint|TestGetFiles_DuplicateFingerprintAllocBudget)$' -count=1`
- `go test ./filebeat/input/filestream -run '^TestGetFiles' -count=1`
- Behavior remains unchanged; optimization only gates duplicate-tracking map usage to overlap-capable path sets.
## Evidence
- Benchmark command (same before/after):
- `BENCH_TREE_FILES=20000 go test -run '^$' -bench '^(BenchmarkGetFilesSelective|BenchmarkGetFilesExcludeMost|BenchmarkGetFilesManyPatterns|BenchmarkGetFilesLiteralMidComponent|BenchmarkGetFilesMixed)$' -benchmem -count=3 ./filebeat/input/filestream`
- Relevant code region:
- `filebeat/input/filestream/fscanner.go:309-323`
- `filebeat/input/filestream/fscanner.go:345-352`
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/32737306027)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Aug 31, 2026, 2:28 PM UTC
Contributor guide
Research direction
Start in filebeat/input/filestream/fscanner.go at the newScanState and process regions around lines 309-323 and 345-352. Compare duplicate tracking for overlapping and non-overlapping path sets, then run the listed TestGetFiles tests and benchmark command. Done means behavior remains unchanged while the non-overlap benchmark avoids unnecessary duplicate-map work and shows the reported allocation and timing improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100