Tracking the performance of `-Zchecksum-freshness`
Open
Nobody has claimed this yet.
A-rebuild-detection
C-tracking-issue
Performance
Z-checksum-freshness
- Dominant language
- Rust
- Stars
- 15.5k
- Forks
- 3k
- Avg merge
- 23h 30m
- Merged PRs (30d)
- 51
Description
This issue is intended for tracking performance related discussion and benchmarks for #14136. #14136 itself is for overall progress reports.
Useful cargo script to summarize number and size of files got checksum'd
#!/usr/bin/env cargo
---cargo
[dependencies]
walkdir = "2"
---
use std::collections::HashMap;
use std::ffi::OsStr;
use std::fs::File;
use std::io::prelude::*;
use std::io::BufReader;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let walker = walkdir::WalkDir::new("target/debug/deps")
.into_iter()
.filter_entry(|e| e.file_type().is_dir() || e.path().extension() == Some(OsStr::new("d")));
let mut map = HashMap::with_capacity(8192);
for entry in walker {
let entry = entry?;
if entry.file_type().is_dir() {
continue;
}
for line in BufReader::new(File::open(entry.path())?).lines() {
let line = line?;
let Some(rest) = line.strip_prefix("# checksum:") else {
continue;
};
let mut parts = rest.splitn(3, ' ');
let (_, size, path) = (parts.next(), parts.next().unwrap(), parts.next().unwrap());
let size: u64 = size.strip_prefix("file_len:").unwrap().parse()?;
if let Some(old_size) = map.insert(path.to_string(), size) {
assert_eq!(old_size, size, "size doesn't match for {path}: {old_size} vs {size}");
}
}
}
let mut stdout = std::io::stdout().lock();
writeln!(stdout, "number of files: {}", map.len())?;
writeln!(stdout, "total file size: {}", map.values().sum::<u64>())?;
Ok(())
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
This issue tracks performance discussion and benchmarks for #14136 rather than naming a specific change. Start by reviewing #14136 and the supplied Cargo script, which summarizes checksum metadata under target/debug/deps. Done is not defined beyond contributing performance findings or benchmarks to the discussion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- build-system, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100