fish-shell / fish-shell/fish-shell

`while read` has up to 2x overhead over buffer-and-loop

Open
#4,972 5 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
34.2k
Forks
2.4k
Avg merge
2d 8h
Merged PRs (30d)
13

Description

From when I was working on the new `read` modes for splitting by line, etc. I suspected there was a hidden performance problem with `read` because it involves opening the fd each time it is invoked.

I'm not sure if there's anything we can do about it, but `while read ...` is around 2x slower for larger inputs as compared to separate buffering and looping. Some of this is inherit in the additional execution of an extra builtin in the loop (`while read` has to execute `read` each time, but `for ... in ...` doesn't have that overhead), but the overhead shouldn't be enough to account for the performance difference I'm seeing.

This sucks because of the limits on content buffering, which _requires_ the usage of `read` to handle large content.. but thereby imposes the performance penalty.

Obviously one (messy) option is to keep the handle open and then have to worry about when it should be closed (after the command executes fully?) but that can introduce a lot of hidden problems if the command executed interfaces in any way with the source feeding into `read`.

Another option is for `while read` to be a new, _single_ builtin, which is an evil hack but much cleaner...

```fish
~> hyperfine -i ./read-by-line.fish ./read-at-once.fish
Benchmark #1: ./read-by-line.fish

Time (mean ± σ): 10.634 s ± 0.462 s [User: 7.701 s, System: 3.507 s]

Range (min … max): 10.035 s … 11.285 s

Warning: Ignoring non-zero exit code.

Benchmark #2: ./read-at-once.fish

Time (mean ± σ): 4.857 s ± 0.164 s [User: 4.225 s, System: 0.624 s]

Range (min … max): 4.688 s … 5.260 s

Summary

'./read-at-once.fish' ran
2.19x faster than './read-by-line.fish'
```

for the scripts

read-by-line.fish:
```fish
#!/usr/bin/env fish

cat /usr/share/dict/words | while read -lL line
echo $line
end
```

and

read-at-once.fish:
```fish
#!/usr/bin/env fish

set -l lines (cat /usr/share/dict/words)
for line in $lines
echo $line
end
```

When reading smaller content (only 50 lines), reading at once is 22% faster (accounting for shell startup overhead).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.