rust-lang / rust-lang/mdBook

[Performance] Copying files with `utils::fs::copy_files_except_ext` is slow

Open
#2,348 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

C-enhancement
Dominant language
Rust
Stars
22.2k
Forks
1.9k
PR merge metrics
PR metrics pending

Description

Problem

Our book is growing and by now building it takes approx. 1m30s. Most of the time is spent in HtmlHandlebars::render when copying files from src to dest.

In a small proof-of-concept I replaced the call to copy_files_except_ext locally with rsync. Execution time dropped to 8s.

Proposed Solution

The copy_files_except_ext seems to be a bottleneck. Various performance improvements can be thought of:

  • Parallelize, e.g., by starting threads for copying sub-directories (though not trivial to fine-tune for performance, e.g., limit to CPU cores or the like,...)
  • Use platform-dependent tools like rsync if available. Very fast, allows for exclude patterns.
Notes

I've seen the comment on recursion while copying. I did not dig into that but is it really an issue? Only if the target directory is a sub-directory of source or if we encounter a (sym-)link into the target directory somewhere in source, I would assume. Both would be strange setups, wouldn't it?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating utils::fs::copy_files_except_ext and its use from HtmlHandlebars::render. Reproduce the reported build timing, compare it with the rsync proof of concept, and investigate the existing recursion comment. Done means file copying is measurably faster without breaking the stated source-to-destination behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.