elastic / elastic/beats

[performance-profiler] Skip transform loop for NOP codec in readfile LineReader decode

Open
#49,946 3 comments 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

## Hot Path
`LineReader.decode` in `libbeat/reader/readfile/line.go` is hot for plain-text inputs (NOP codec), but still runs the generic `Transform` loop and per-chunk copying path.

Relevant locations:
- `libbeat/reader/readfile/line.go:304-337` (`(*LineReader).decode`)
- `libbeat/reader/readfile/line.go:57-85` (`NewLineReader` initialization)

## Profiling Data
**Before:**
```text
go test -run=^$ -bench='BenchmarkEncoderReader/long_lines$' -benchmem -benchtime=500ms -count=3 ./libbeat/reader/readfile

BenchmarkEncoderReader/long_lines-4 417 1368752 ns/op 5095769 B/op 773 allocs/op
BenchmarkEncoderReader/long_lines-4 471 1325137 ns/op 5095768 B/op 773 allocs/op
BenchmarkEncoderReader/long_lines-4 453 1326205 ns/op 5095765 B/op 773 allocs/op
```

## Proposed Change
Detect the NOP decoder once in `NewLineReader` and fast-path `decode` to write the full slice directly, bypassing the per-iteration `Transform` loop:

```diff
+ nopDecoder bool
...
+ decoder := config.Codec.NewDecoder()
+ nopDecoder: decoder.Transformer == transform.Nop,
...
+ if r.nopDecoder {
+ _, err = r.outBuffer.Write(inBytes[:end])
+ r.byteCount += end
+ return end, err
+ }
```

## Results
**After:**
```text
go test -run=^$ -bench='BenchmarkEncoderReader/long_lines$' -benchmem -benchtime=500ms -count=3 ./libbeat/reader/readfile

BenchmarkEncoderReader/long_lines-4 819 697723 ns/op 2945374 B/op 373 allocs/op
BenchmarkEncoderReader/long_lines-4 837 687463 ns/op 2945364 B/op 373 allocs/op
BenchmarkEncoderReader/long_lines-4 867 751050 ns/op 2945357 B/op 373 allocs/op
```

**Improvement (mean):**
- Time: **~46.9% faster** (1,340,031 ns/op -> 712,079 ns/op)
- Memory: **~42.2% lower** (5,095,767 B/op -> 2,945,365 B/op)
- Allocations: **~51.7% fewer** (773 -> 373 allocs/op)

## Verification
- `go test ./libbeat/reader/readfile ./libbeat/common/streambuf` passed.
- Benchmark command is identical before/after and outputs are demonstrably different.
- Change is behavior-preserving (NOP codec shortcut only; non-NOP path unchanged).

## Evidence
Commands used:
- `go test -run=^$ -bench='BenchmarkEncoderReader/long_lines$' -benchmem -benchtime=500ms -count=3 ./libbeat/reader/readfile` (before and after)
- `go test ./libbeat/reader/readfile ./libbeat/common/streambuf`

Checked for likely duplicates against previously filed profiler issues (`/tmp/previous-findings.json`); this specific NOP decode fast path was not listed.

> [!NOTE]
>
> 🔒 Integrity filtering filtered 3 items
>
> Integrity filtering activated and filtered the following items during workflow execution.
> This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.
>
> - issue:elastic/beats#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - [#48986](https://github.com/elastic/beats/issues/48986) (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - [#48991](https://github.com/elastic/beats/issues/48991) (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/24035408116)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Apr 13, 2026, 2:36 PM UTC

Contributor guide

Open the contributing guide

Research direction

Start with libbeat/reader/readfile/line.go, especially NewLineReader at lines 57-85 and LineReader.decode at lines 304-337. Run the provided long_lines benchmark and the readfile and streambuf tests before and after the change. Done means the NOP path is optimized while non-NOP decoding remains unchanged and the verification commands pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.