FuelLabs / FuelLabs/fuelup

Concurrency issue in `hardlink()` when running `fuelup` in parallel

Open
#833 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
282
Forks
150
Avg merge
2d 17h
Merged PRs (30d)
2

Description

This is the result of the analysis of failing o2-exports tests done by Claude Code Fable 5. When running `cargo test -all` on tests they were randomly failing with the message:

```
fuel-core process exited before reporting its bound address
```

The root cause of the issue id fuelup proxy losing a hardlink race while launching fuel-core in parallel (due to parallel test execution).

What I did

I wrapped fuel-core in a script that tees the child's stdout/stderr to files (the fuels-rs error path kills the child without dumping its logs, so the real error was being swallowed), then reproduced the flake. The failed process exited with status 0, wrote nothing to stderr, and printed this to stdout:

Could not create link: /home/kebradalaonda/.fuelup/store/fuel-core-0.48.0/fuel-core-keygen->/home/kebradalaonda/.fuelup/toolchains/testnet-x86_64-unknown-linux-gnu/bin/fuel-core-keygen

Root cause chain

1. ~/.fuelup/bin/fuel-core is not fuel-core — it's a hardlink to the fuelup binary acting as a proxy.
2. Your repo has a fuel-toolchain.toml pinning fuel-core = "0.48.0" on the testnet channel. Because of that version pin, fuelup's proxy_cli.rs calls Toolchain::add_component() on every single invocation.
3. add_component (fuelup src/toolchain.rs), even when the component is already fully installed, unconditionally re-creates the store→toolchain hardlinks. And hardlink() in src/file.rs is remove_file(link) then hard_link(...) — a textbook TOCTOU race.
4. Parallel tests each spawn "fuel-core" → multiple proxies relink concurrently → the loser's hard_link hits EEXIST (the symlink fallback fails the same way), the proxy prints the error to stdout and exits 0 without exec'ing fuel-core. fuels-rs sees stderr EOF and reports "fuel-core process exited before reporting its bound address".

Fix

FuelLabs/fuelup add_component should skip relinking when the link already points at the right inode (or take a file lock); the proxy also shouldn't report this failure on stdout with exit code 0. Note the race can even momentarily delete the toolchain's fuel-core binary itself, so other flake modes are possible.

Additional finding

Concurrent proxies sometimes conclude the component is missing entirely and start re-downloading fuel-core from GitHub mid-test-run, then collide in the store directory. That explains the store re-extraction timestamp we saw earlier, and it's worth including in the fuelup issue: the proxy both relinks unconditionally and can trigger redundant re-installs under concurrency.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with src/toolchain.rs, src/file.rs, and proxy_cli.rs, then run the affected parallel tests with cargo test -all to reproduce the fuel-core launch failure. Trace add_component and hardlink during concurrent proxy launches; done means parallel invocations no longer race during relinking or redundant installation, and a proxy failure is reported through the appropriate failure status and output.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.