rust-lang / rust-lang/cargo

Tracking the performance of `-Zchecksum-freshness`

Open
#14,722 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-rebuild-detection C-tracking-issue Performance Z-checksum-freshness
Dominant language
Rust
Stars
15.5k
Forks
3k
Avg merge
23h 30m
Merged PRs (30d)
51

Description

This issue is intended for tracking performance related discussion and benchmarks for #14136. #14136 itself is for overall progress reports.

Useful cargo script to summarize number and size of files got checksum'd

#!/usr/bin/env cargo
---cargo
[dependencies]
walkdir = "2"
---

use std::collections::HashMap;
use std::ffi::OsStr;
use std::fs::File;
use std::io::prelude::*;
use std::io::BufReader;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let walker = walkdir::WalkDir::new("target/debug/deps")
        .into_iter()
        .filter_entry(|e| e.file_type().is_dir() || e.path().extension() == Some(OsStr::new("d")));

    let mut map = HashMap::with_capacity(8192);

    for entry in walker {
        let entry = entry?;
        if entry.file_type().is_dir() {
            continue;
        }
        for line in BufReader::new(File::open(entry.path())?).lines() {
            let line = line?;
            let Some(rest) = line.strip_prefix("# checksum:") else {
                continue;
            };
            let mut parts = rest.splitn(3, ' ');
            let (_, size, path) = (parts.next(), parts.next().unwrap(), parts.next().unwrap());
            let size: u64 = size.strip_prefix("file_len:").unwrap().parse()?;
            if let Some(old_size) = map.insert(path.to_string(), size) {
                assert_eq!(old_size, size, "size doesn't match for {path}: {old_size} vs {size}");
            }
        }
    }

    let mut stdout = std::io::stdout().lock();
    writeln!(stdout, "number of files: {}", map.len())?;
    writeln!(stdout, "total file size: {}", map.values().sum::<u64>())?;

    Ok(())
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

This issue tracks performance discussion and benchmarks for #14136 rather than naming a specific change. Start by reviewing #14136 and the supplied Cargo script, which summarizes checksum metadata under target/debug/deps. Done is not defined beyond contributing performance findings or benchmarks to the discussion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.