[Performance] Copying files with `utils::fs::copy_files_except_ext` is slow
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 22.2k
- Forks
- 1.9k
- PR merge metrics
- PR metrics pending
Description
Problem
Our book is growing and by now building it takes approx. 1m30s. Most of the time is spent in HtmlHandlebars::render when copying files from src to dest.
In a small proof-of-concept I replaced the call to copy_files_except_ext locally with rsync. Execution time dropped to 8s.
Proposed Solution
The copy_files_except_ext seems to be a bottleneck. Various performance improvements can be thought of:
- Parallelize, e.g., by starting threads for copying sub-directories (though not trivial to fine-tune for performance, e.g., limit to CPU cores or the like,...)
- Use platform-dependent tools like
rsyncif available. Very fast, allows for exclude patterns.
Notes
I've seen the comment on recursion while copying. I did not dig into that but is it really an issue? Only if the target directory is a sub-directory of source or if we encounter a (sym-)link into the target directory somewhere in source, I would assume. Both would be strange setups, wouldn't it?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating utils::fs::copy_files_except_ext and its use from HtmlHandlebars::render. Reproduce the reported build timing, compare it with the rsync proof of concept, and investigate the existing recursion comment. Done means file copying is measurably faster without breaking the stated source-to-destination behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100