rust-lang / rust-lang/git2-rs

Sudden ~10 min Delay in `RepoBuilder::clone` Without `Code/Dependency` Changes

Open
#1,158 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
2.1k
Forks
450
Avg merge
11m
Merged PRs (30d)
1

Description

Sudden Performance Degradation in Async RepoBuilder::clone (approx. 10 min delay)

Problem Description:

I am experiencing a significant and sudden performance degradation when cloning repositories using git2-rs. An operation that previously completed quickly now takes approximately 10 minutes. This slowdown started occurring recently without any intentional changes to my code or explicit dependency updates (Cargo.lock appears unchanged relative to when it was working fast).

Cloning the same repository using command-line git or downloading related resources via curl with the same token remains fast, suggesting the issue is specific to the git2-rs interaction or its underlying libgit2 usage in my setup, potentially related to the async context or authentication.

Environment:

  • git2-rs version: [0.20.1]
  • libgit2-sys version: [0.18.1+1.9.0]
  • Rust Version: rustc 1.86.0**]
  • Operating System: [e.g., Linux Ubuntu 22.04]

Code Snippet:

Here is the relevant function performing the clone:

fn test_git2(repo_url: &str, dest_path: &Path, token: Option<String>) {
    println!("\n=== Testing git2 library (with detailed progress) ===");

    // Optional: Clean up destination directory before cloning if it exists
    // This prevents errors if the directory exists and is not empty,
    // especially since CheckoutBuilder.force() might be needed.
    if dest_path.exists() {
        println!("Removing existing directory: {}", dest_path.display());
        if let Err(e) = std::fs::remove_dir_all(dest_path) {
            println!("Warning: Failed to remove existing directory {}: {}", dest_path.display(), e);
            // Decide if you want to proceed or return an error here
        }
    }

    let start = Instant::now();

    let result = clone_with_git2(repo_url, dest_path, token);

    let elapsed = start.elapsed();

    match result {
        Ok(_) => println!("\ngit2 clone successful in {:.4} seconds", elapsed.as_secs_f64()), // Added newline for cleaner output
        Err(e) => println!("\ngit2 clone failed: {:?}", e), // Added newline
    }
}

fn clone_with_git2(repo_url: &str, dest_path: &Path, token: Option<String>) -> Result<Repository, git2::Error> {

    // --- 1. Setup Fetch Options ---
    let mut fetch_options = FetchOptions::new();
    fetch_options.update_fetchhead(false); // Don't update FETCH_HEAD (minor optimization)
    fetch_options.depth(1); // Shallow clone - only get the latest commit

    // --- 2. Setup Remote Callbacks (Authentication & Fetch Progress) ---
    let mut callbacks = RemoteCallbacks::new();

    // Detailed Transfer (Fetch) Progress Reporting
    callbacks.transfer_progress(|progress: Progress| {
        let network_pct = if progress.total_objects() > 0 {
            (100 * progress.received_objects()) / progress.total_objects()
        } else { 0 };
        let index_pct = if progress.total_objects() > 0 {
            (100 * progress.indexed_objects()) / progress.total_objects()
        } else { 0 };
        let kbytes = progress.received_bytes() / 1024;

        // Print progress information without a newline, using '\r' to overwrite the line
        print!(
            "Fetch Progress: net {:3}% ({:4} kb, {:>5}/{:<5})  /  idx {:3}% ({:>5}/{:<5})\r",
            network_pct,
            kbytes,
            progress.received_objects(),
            progress.total_objects(),
            index_pct,
            progress.indexed_objects(),
            progress.total_objects()
        );
        // Flush stdout to ensure the progress is displayed immediately
        io::stdout().flush().unwrap();
        true // Return true to continue the transfer
    });

    // Authentication Callback (if token is provided)
    if let Some(token_str) = token {
        // Clone the token string so it can be moved into the closure
        let token_clone = token_str.clone();
        callbacks.credentials(move |_, _, _| {
            // This closure will be called by libgit2 when authentication is needed
             // println!("Credentials callback invoked"); // Uncomment for debugging auth calls
            Cred::userpass_plaintext("oauth2", &token_clone)
        });
    }

    // Apply the callbacks to the fetch options
    fetch_options.remote_callbacks(callbacks);

    // --- 3. Setup Checkout Builder (Checkout Progress) ---
    let mut checkout_builder = CheckoutBuilder::new();
    checkout_builder.force(); // Force checkout to overwrite files if they exist
                                // Equivalent to `git clone --force` behavior.
                                // Remove if you want it to fail if the directory has conflicting files.

    // Detailed Checkout Progress Reporting
    checkout_builder.progress(|path, current, total| {
        let path_str = path.map(|p| p.to_string_lossy()).unwrap_or_default();
        // Print progress, overwriting the line with '\r'
        print!("Checkout Progress: [{}/{}] {}\r", current, total, path_str);
        // Flush stdout to ensure the progress is displayed immediately
        io::stdout().flush().unwrap();
    });


    // --- 4. Setup Repo Builder (combines fetch and checkout) ---
    let mut repo_builder = RepoBuilder::new();
    repo_builder.bare(false); // Ensure it's not a bare repo (we want a working directory)
    repo_builder.fetch_options(fetch_options); // Use the fetch options configured above
    repo_builder.with_checkout(checkout_builder); // Use the checkout builder configured above

    println!("Starting git2 clone (shallow, depth=1) from {} to {}", repo_url, dest_path.display());

    // --- 5. Perform the Clone ---
    // This call will use the configured fetch_options (including callbacks)
    // and the configured checkout_builder (including its progress callback).
    let result = repo_builder.clone(repo_url, dest_path);

    // Print a newline character after the progress indicators are done to avoid messing up subsequent output
    println!();

    result // Return the result of the clone operation
}

Could there be any known issues, potential deadlocks, or changes in recent libgit2 or git2-rs versions that might cause such a delay, especially when interacting with authentication callbacks or specific server responses within an async context (even if correctly using spawn_blocking)? Are there recommended steps for diagnosing network or callback-related hangs within libgit2 from git2-rs (e.g., enabling libgit2 tracing)?

Adding detailed progress callbacks (for fetch and checkout, as shown in other examples) might provide more granular clues, but I wanted to report the sudden change and the potential async interaction issue first.

Thank you for your time and insights!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository source file or test is named. Start at the provided RepoBuilder::clone and clone_with_git2 reproducer, comparing git2-rs timing with command-line git while observing fetch, checkout, and authentication callbacks. Done means isolating a reproducible cause for the delay or documenting that it depends on the environment, server response, or async usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, rust
Domain
networking, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.