rust-lang / rust-lang/rustup

Broken Cargo binaries on Windows Docker images due to symlinks

Open
#4,291 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7k
Forks
1.1k
Avg merge
22h 40m
Merged PRs (30d)
46

Description

Verification
Problem

Since 1.28.0, we have been finding subtle yet impossible to debug issues in our CI, where we get nonsensical results like this when trying to call Cargo:

Traceback (most recent call last):
  File "<string>", line 24, in <module>
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\main.py", line 224, in main
    Main(sys.argv[1:])
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\main.py", line 52, in __init__
    self.run_command()
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\main.py", line 192, in run_command
    res = commands.run(command, self.config, self.args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\commands\__init__.py", line 79, in run
    return _commands[command].run(config, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\commands\bootstrap.py", line 179, in run
    run_until_complete(build_tools_task)
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\utils\__init__.py", line 763, in run_until_complete
    result = loop.run_until_complete(tasks)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\bootstrap\build_tools.py", line 145, in fetch_recipes
    await Fetch.fetch(self.cookbook, self.recipes, False, False, False, False, jobs)
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\commands\fetch.py", line 112, in fetch
    await fetch_print_wrapper(recipe.name, stepfunc)
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\commands\fetch.py", line 100, in fetch_print_wrapper
    await stepfunc()
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\build\recipe.py", line 102, in async_wrapped
    ret = await stepfunc(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\build\source.py", line 456, in fetch
    await self.extract_impl(fetching=True)
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\build\source.py", line 487, in extract_impl
    await self.cargo_vendor(not fetching or self.offline)
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\build\source.py", line 137, in cargo_vendor
    ct = await shell.async_call_output(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\utils\shell.py", line 345, in async_call_output
    raise e
  File "C:\Users\Administrator\runner\builds\amyspark\cerbero\cerbero\utils\shell.py", line 328, in async_call_output
    proc = await asyncio.create_subprocess_exec(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\base_events.py", line 1694, in subprocess_exec
    transport = await self._make_subprocess_transport(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\windows_events.py", line 399, in _make_subprocess_transport
    transp = _WindowsSubprocessTransport(self, protocol, args, shell,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "C:\Python311\Lib\asyncio\windows_events.py", line 921, in _start
    self._proc = windows_utils.Popen(
                 ^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\windows_utils.py", line 153, in __init__
    super().__init__(args, stdin=stdin_rfd, stdout=stdout_wfd,
  File "C:\Python311\Lib\subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Python311\Lib\subprocess.py", line 1493, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 3] The system cannot find the path specified
C:/Users/Administrator/runner/builds/amyspark/cerbero/cb/rust/cargo/bin/cargo.exe vendor C:/Users/Administrator/runner/builds/amyspark/cerbero/cerbero-sources/bindgen-cli-0.71.1/cargo-vendor C:/Users/Administrator/runner/builds/amyspark/cerbero/cb/sources/build-tools/bindgen-cli-0.71.1
C:/Users/Administrator/runner/builds/amyspark/cerbero/cb/rust/cargo/bin/cargo.exe True
['cargo-clippy.exe', 'cargo-fmt.exe', 'cargo-miri.exe', 'cargo.exe', 'clippy-driver.exe', 'rls.exe', 'rust-analyzer.exe', 'rust-gdb.exe', 'rust-gdbgui.exe', 'rust-lldb.exe', 'rustc.exe', 'rustdoc.exe', 'rustfmt.exe', 'rustup.exe']

Where Cargo clearly exists, yet is impossible to be executed. It's simple to notice why, once the files are captured:

Image

I tracked down the regression to this: rustup uses symlinks on Windows: https://github.com/rust-lang/rustup/issues/3677

Steps
  1. Install Rust on a Dockerized Windows environment, via rustup-init. Use multiple targets (in our case i386 and x86-64, both windows-msvc and windows-gnu)
  2. Call any Cargo command
Possible Solution(s)

Revert the commit https://github.com/rust-lang/rustup/pull/3687/commits/7e644c6901c030ba2ccb248f891198f0c55bd05f.

Notes

Build log here: https://gitlab.freedesktop.org/amyspark/cerbero/-/jobs/74755751

Rustup version
(We don't log `rustup --version` but I can verify it's 1.28.1 for x86_64-pc-windows-msvc)
Installed toolchains
(Same as above: stable-x86_64-pc-windows-msvc)
OS version
Windows Server LTSC 2022

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the rustup-init installation in a Dockerized Windows environment with the listed multiple targets, then inspect the regression commit linked from the issue and the Windows symlink behavior. Done means Cargo commands execute successfully from the installed cargo/bin directory in that environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
devtools, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.