Sort tempfile logic opens up extra files
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 24.1k
- Forks
- 2k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 365
Description
As part of investigating this issue, found that we're using extra file descriptors to mange our tempfile logic for batched-sort operations. This is a problem because some of the GNU compatibility tests run with ulimit enforcing strict limits on the number of files that sort can open.
Looking at the open files when starting a batched sort, I see...
$ ls -l /proc/1778302/fd
total 0
lrwx------ 1 karl karl 64 Dec 8 19:43 0 -> /dev/pts/3
l-wx------ 1 karl karl 64 Dec 8 19:43 1 -> /home/karl/Rust/gnu/tests/sort/sort-merge-fdlimit.log
l-wx------ 1 karl karl 64 Dec 8 19:43 2 -> /home/karl/Rust/gnu/gt-sort-merge-fdlimit.sh.A1OJ/err/merge-random-err
lr-x------ 1 karl karl 64 Dec 8 19:43 3 -> 'pipe:[3021894]'
l-wx------ 1 karl karl 64 Dec 8 19:43 4 -> 'pipe:[3021894]'
It's the last two of these that are problematic. These two pipes are created when we set up the Tempfile logic...
fn init_tmp_dir(&mut self) -> UResult<()> {
<snip>
let path = self.temp_dir.as_ref().unwrap().path().to_owned();
let lock = self.lock.clone();
// *** The line below will open the two pipes.
ctrlc::set_handler(move || {
// Take the lock so that `next_file_path` returns no new file path,
// and the program doesn't terminate before the handler has finished
let _lock = lock.lock().unwrap();
if let Err(e) = remove_tmp_dir(&path) {
show_error!("failed to delete temporary directory: {}", e);
}
std::process::exit(2)
})
.map_err(|e| USimpleError::new(2, format!("failed to set up signal handler: {e}")))
}
To fix this, I propose to just simplify this logic and just remove the Tempdir altogether. Just put any temporary files directly into /tmp (which is what GNU-sort does) and have the OS do the cleanup automatically rather than create a dedicated folder for them. We can then remove this code above that uses the ctrlc crate.
I don't know the history on this, so maybe there's a good reason to use a separate folder - If so, please let me know!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at sort's init_tmp_dir logic and trace how Tempfile, the temporary directory, and the ctrlc handler are used during batched sorts. Run the GNU compatibility tests for sort-merge-fdlimit and inspect open descriptors while sorting. Done means the extra pipes are no longer opened and the fd-limit tests still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100