BurntSushi / BurntSushi/encoding_rs_io

DecodeReaderBytes should implement read_to_end

Open
#16 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
30
Forks
10
PR merge metrics
No merged PRs in 30d

Description

Currently `rg` with `--multiline` when operating on many files can be 50x slower without `--mmap` compared to `--multiline --mmap`.

More than 99% of CPU time is spent in `ReadBuf::initialize_unfilled`, which is called from `default_read_buf`, called from `default_read_to_end`, called from `read_to_end` here:

```rust
if self.config.heap_limit.is_none() {
let mut buf = self.multi_line_buffer.borrow_mut();
buf.clear();
let cap =
file.metadata().map(|m| m.len() as usize + 1).unwrap_or(0);
buf.reserve(cap);
read_from.read_to_end(&mut *buf).map_err(S::Error::error_io)?;
return Ok(());
}
```

https://github.com/BurntSushi/ripgrep/blob/master/crates/searcher/src/searcher/mod.rs#L911-L919

If `buf` grows large, then the `initialize_unfilled` function will clear the entire capacity of the vector for every file, irrespective of the file's size, which in my case resulted in 300 GB of memory transfers for only 3 GB of data.

If `DecodeReaderBytes` implemented the `read_to_end` function, then it would be able to avoid initializing the entire buffer, only writing to the part of it that actually needs to be written.

(Alternatively, `ripgrep` could be changed to not call `read_to_end`, or to not reuse a single `Vec` for every file.)

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate DecodeReaderBytes and its read_to_end entry point, then compare it with the call in crates/searcher/src/searcher/mod.rs at lines 911-919. Done means the adapter avoids initializing the entire reused buffer while preserving decoded reads and error handling; no repository test is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.