elastic / elastic/beats

[performance-profiler] Avoid duplicate filename map work in filestream GetFiles non-overlap scans

Open Beginner friendly
#52,808 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

## Hot Path
`filebeat/input/filestream/fscanner.go:309-323` and `filebeat/input/filestream/fscanner.go:345-352` in `newScanState`/`process`.

`process` currently checks `uniqueFiles` for every matched file. For non-overlapping path sets, that duplicate-filter map is unnecessary work and allocation pressure.

## Profiling Data
**Before:**
```text
Command:
BENCH_TREE_FILES=20000 go test -run '^$' -bench '^(BenchmarkGetFilesSelective|BenchmarkGetFilesExcludeMost|BenchmarkGetFilesManyPatterns|BenchmarkGetFilesLiteralMidComponent|BenchmarkGetFilesMixed)$' -benchmem -count=3 ./filebeat/input/filestream

BenchmarkGetFilesExcludeMost
105087054 ns/op 15961360 B/op 232235 allocs/op
123219598 ns/op 15978988 B/op 232249 allocs/op
125129221 ns/op 15974268 B/op 232245 allocs/op
```

## Proposed Change
Allocate and use `uniqueFiles` only when overlap is possible (`s.pathsCanOverlap`), and skip duplicate-file map checks otherwise.

```diff
--- a/filebeat/input/filestream/fscanner.go
+++ b/filebeat/input/filestream/fscanner.go
@@
func (s *fileScanner) newScanState(opts loginp.FileScanOptions) *scanState {
+ var uniqueFiles map[string]struct{}
+ if s.pathsCanOverlap {
+ uniqueFiles = make(map[string]struct{}, s.lastCount)
+ }
@@
- uniqueFiles: make(map[string]struct{}, s.lastCount),
+ uniqueFiles: uniqueFiles,
@@
- if _, knownFile := st.uniqueFiles[filename]; knownFile {
- st.metrics.FilesNoIngestTarget++
- return
- }
- st.uniqueFiles[filename] = struct{}{}
+ if st.uniqueFiles != nil {
+ if _, knownFile := st.uniqueFiles[filename]; knownFile {
+ st.metrics.FilesNoIngestTarget++
+ return
+ }
+ st.uniqueFiles[filename] = struct{}{}
+ }
```

## Results
**After:**
```text
Command:
BENCH_TREE_FILES=20000 go test -run '^$' -bench '^(BenchmarkGetFilesSelective|BenchmarkGetFilesExcludeMost|BenchmarkGetFilesManyPatterns|BenchmarkGetFilesLiteralMidComponent|BenchmarkGetFilesMixed)$' -benchmem -count=3 ./filebeat/input/filestream

BenchmarkGetFilesExcludeMost
90112158 ns/op 13913980 B/op 232062 allocs/op
91235473 ns/op 14205879 B/op 232080 allocs/op
98174854 ns/op 14201968 B/op 232076 allocs/op
```

**Improvement:** mean `ns/op` improved **20.91%**; mean `B/op` improved **11.67%** for `BenchmarkGetFilesExcludeMost`.

## Verification
- `go test ./filebeat/input/filestream -run '^(TestGetFiles_GrowingRawSuppression|TestGetFiles_DuplicateFingerprint|TestGetFiles_DuplicateFingerprintAllocBudget)$' -count=1`
- `go test ./filebeat/input/filestream -run '^TestGetFiles' -count=1`
- Behavior remains unchanged; optimization only gates duplicate-tracking map usage to overlap-capable path sets.

## Evidence
- Benchmark command (same before/after):
- `BENCH_TREE_FILES=20000 go test -run '^$' -bench '^(BenchmarkGetFilesSelective|BenchmarkGetFilesExcludeMost|BenchmarkGetFilesManyPatterns|BenchmarkGetFilesLiteralMidComponent|BenchmarkGetFilesMixed)$' -benchmem -count=3 ./filebeat/input/filestream`
- Relevant code region:
- `filebeat/input/filestream/fscanner.go:309-323`
- `filebeat/input/filestream/fscanner.go:345-352`

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/32737306027)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Aug 31, 2026, 2:28 PM UTC

Contributor guide

Open the contributing guide

Research direction

Start in filebeat/input/filestream/fscanner.go at the newScanState and process regions around lines 309-323 and 345-352. Compare duplicate tracking for overlapping and non-overlapping path sets, then run the listed TestGetFiles tests and benchmark command. Done means behavior remains unchanged while the non-overlap benchmark avoids unnecessary duplicate-map work and shows the reported allocation and timing improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.