fish-shell / fish-shell/fish-shell
`while read` has up to 2x overhead over buffer-and-loop
- Dominant language
- Rust
- Stars
- 34.2k
- Forks
- 2.4k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 13
Description
From when I was working on the new `read` modes for splitting by line, etc. I suspected there was a hidden performance problem with `read` because it involves opening the fd each time it is invoked.
I'm not sure if there's anything we can do about it, but `while read ...` is around 2x slower for larger inputs as compared to separate buffering and looping. Some of this is inherit in the additional execution of an extra builtin in the loop (`while read` has to execute `read` each time, but `for ... in ...` doesn't have that overhead), but the overhead shouldn't be enough to account for the performance difference I'm seeing.
This sucks because of the limits on content buffering, which _requires_ the usage of `read` to handle large content.. but thereby imposes the performance penalty.
Obviously one (messy) option is to keep the handle open and then have to worry about when it should be closed (after the command executes fully?) but that can introduce a lot of hidden problems if the command executed interfaces in any way with the source feeding into `read`.
Another option is for `while read` to be a new, _single_ builtin, which is an evil hack but much cleaner...
```fish
~> hyperfine -i ./read-by-line.fish ./read-at-once.fish
Benchmark #1: ./read-by-line.fish
Time (mean ± σ): 10.634 s ± 0.462 s [User: 7.701 s, System: 3.507 s]
Range (min … max): 10.035 s … 11.285 s
Warning: Ignoring non-zero exit code.
Benchmark #2: ./read-at-once.fish
Time (mean ± σ): 4.857 s ± 0.164 s [User: 4.225 s, System: 0.624 s]
Range (min … max): 4.688 s … 5.260 s
Summary
'./read-at-once.fish' ran
2.19x faster than './read-by-line.fish'
```
for the scripts
read-by-line.fish:
```fish
#!/usr/bin/env fish
cat /usr/share/dict/words | while read -lL line
echo $line
end
```
and
read-at-once.fish:
```fish
#!/usr/bin/env fish
set -l lines (cat /usr/share/dict/words)
for line in $lines
echo $line
end
```
When reading smaller content (only 50 lines), reading at once is 22% faster (accounting for shell startup overhead).
Contributor guide
Assessment
This issue has not been assessed yet.