indygreg / indygreg/PyOxidizer

The pip_install step copies the entire directory tree, including repo data

Open
#342 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

It's mostly a small issue I noticed when building Mercurial and switching back and forth between MSYS and WSL. The WSL build of Mercurial generates the `.hg/wcache/checklink` symlink that causes PyOxidizer to fail on the Windows side:

```
$ make -C .. pyoxidizer
make: Entering directory `/c/Users/Matt/hg'
pyoxidizer build --path ./rust/hgcli --release
resolving 1 targets
resolving target app_windows
resolving target distribution_windows
resolving target exe_windows
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20201020/cpython-3.8.6-x86_64-pc-windows-msvc-shared-pgo-20201021T0232.tar.zst", sha256: "b842ddc51
a3611b574bd5aba8e233bb9a7a52b0479388187f31723ece09bf898" }
downloading https://github.com/indygreg/python-build-standalone/releases/download/20201020/cpython-3.8.6-x86_64-pc-windows-msvc-shared-pgo-20201021T0232.tar.zst
Python distribution available at c:\Users\Matt\hg\build\pyoxidizer\python_distributions\cpython-3.8.6-x86_64-pc-windows-msvc-shared-pgo-20201021T0232.tar.zst
reading data from Python distribution...
pip installing to C:\Users\Matt\AppData\Local\Temp\pyoxidizer-pip-installIOYIBl\install
Using pip 20.2.4 from c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip (python 3.8)
Non-user install due to --prefix or --target option
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-target-cgeu8f5n
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-ephem-wheel-cache-kguhm92j
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-n0i60uzg
Initialized build tracking at C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-n0i60uzg
Created build tracker: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-n0i60uzg
Entered build tracker: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-n0i60uzg
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-install-2vt21fa2
Processing c:\users\matt\hg
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-req-build-w5r67snc
ERROR: Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\commands\install.py", line 323, in run
requirement_set = resolver.resolve(
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
discovered_reqs.extend(self._resolve_one(requirement_set, req))
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\resolution\legacy\resolver.py", line 340, in _get_abstract_d
ist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\operations\prepare.py", line 467, in prepare_linked_requirem
ent
local_file = unpack_url(
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\operations\prepare.py", line 246, in unpack_url
_copy_source_tree(link.file_path, location)
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\operations\prepare.py", line 186, in _copy_source_tree
shutil.copytree(source, target, **kwargs)
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\shutil.py", line 554, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\shutil.py", line 510, in _copytree
raise Error(errors)
shutil.Error: [('C:\\Users\\Matt\\hg\\.hg\\wcache\\checklink', 'C:\\Users\\Matt\\AppData\\Local\\Temp\\pip-req-build-w5r67snc\\.hg\\wcache\\checklink', "[Errno 22] Invalid argument: 'C:\\\\Users\\\\Matt\\\\hg\\\
\.hg\\\\wcache\\\\checklink'")]
Removed build tracker: 'C:\\Users\\Matt\\AppData\\Local\\Temp\\pip-req-tracker-n0i60uzg'
error[PIP_INSTALL_ERROR]: error running pip install: calling pip install
--> ./rust/hgcli\pyoxidizer.bzl:70:30
|
70 | exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pip_install()

error: error running pip install: calling pip install
make: *** [pyoxidizer] Error 1
make: Leaving directory `/c/Users/Matt/hg'
```

The fix is simply to remove the file on the WSL side, but it's not an immediate failure, so if there's a simple option to tell it to skip `.hg`, `.git`, etc when copying the tree, that would be helpful.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at rust/hgcli/pyoxidizer.bzl line 70, where exe.pip_install() is called, and trace the pip_install implementation that copies the source tree. Reproduce the failure with repository metadata such as .hg/wcache/checklink present. Done means pip_install avoids copying VCS metadata like .hg and .git while retaining the package source tree.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.