rust-lang / rust-lang/rust

Add back io::copy file to pipe optimization

Open
#128,300 19 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

C-feature-request T-libs
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

So to sum up, I think we can add back optimization via:

  • if the file is on squashfs/erofs, then it's immutable (detectable via statfs
  • if the is sealed with F_SEAL_WRITE (prevent modifying of existing contents, shrinking and appending is allowed) and F_SEAL_SEAL (not changing of seals), detectable via fcntl(fd, F_GET_SEALS)
  • if it's on read-only btrfs subvolume (e.g. snapshot), likewise we can do the same for zfs, nilfs, bcachefs
  • If the file is /dev/zero, or from other pseudo fs (e.g. procfs, sysfe), then I think we can assume it's immutable

Otherwise, we'd fallback to:

  • read + write if the file is really small
  • splice to an intermediate pipe with F_MOVE and then splice to the pipe without F_MOVE if the file is more than 1-2 pages
  • if the fs supports reflink and file is really large, then we could create a O_TMPFILE on the system, copy_file_range to it and then splice it

Also, if the file is a named fifo, then just a splice without F_MOVE would work fine.

Originally posted by @NobodyXu in https://github.com/rust-lang/rust/issues/108283#issuecomment-2254439139

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the io::copy implementation and reading the referenced statfs, fcntl, splice, and copy_file_range APIs. Compare the proposed filesystem and file-type cases with the existing behavior; done means restoring the optimization while preserving the listed fallbacks and handling named FIFOs correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
operating-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.