indygreg / indygreg/PyOxidizer
The pip_install step copies the entire directory tree, including repo data
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 6.2k
- Forks
- 256
- PR merge metrics
- No merged PRs in 30d
Description
It's mostly a small issue I noticed when building Mercurial and switching back and forth between MSYS and WSL. The WSL build of Mercurial generates the `.hg/wcache/checklink` symlink that causes PyOxidizer to fail on the Windows side:
```
$ make -C .. pyoxidizer
make: Entering directory `/c/Users/Matt/hg'
pyoxidizer build --path ./rust/hgcli --release
resolving 1 targets
resolving target app_windows
resolving target distribution_windows
resolving target exe_windows
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20201020/cpython-3.8.6-x86_64-pc-windows-msvc-shared-pgo-20201021T0232.tar.zst", sha256: "b842ddc51
a3611b574bd5aba8e233bb9a7a52b0479388187f31723ece09bf898" }
downloading https://github.com/indygreg/python-build-standalone/releases/download/20201020/cpython-3.8.6-x86_64-pc-windows-msvc-shared-pgo-20201021T0232.tar.zst
Python distribution available at c:\Users\Matt\hg\build\pyoxidizer\python_distributions\cpython-3.8.6-x86_64-pc-windows-msvc-shared-pgo-20201021T0232.tar.zst
reading data from Python distribution...
pip installing to C:\Users\Matt\AppData\Local\Temp\pyoxidizer-pip-installIOYIBl\install
Using pip 20.2.4 from c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip (python 3.8)
Non-user install due to --prefix or --target option
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-target-cgeu8f5n
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-ephem-wheel-cache-kguhm92j
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-n0i60uzg
Initialized build tracking at C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-n0i60uzg
Created build tracker: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-n0i60uzg
Entered build tracker: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-n0i60uzg
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-install-2vt21fa2
Processing c:\users\matt\hg
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-req-build-w5r67snc
ERROR: Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\commands\install.py", line 323, in run
requirement_set = resolver.resolve(
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\resolution\legacy\resolver.py", line 183, in resolve
discovered_reqs.extend(self._resolve_one(requirement_set, req))
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\resolution\legacy\resolver.py", line 388, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\resolution\legacy\resolver.py", line 340, in _get_abstract_d
ist_for
abstract_dist = self.preparer.prepare_linked_requirement(req)
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\operations\prepare.py", line 467, in prepare_linked_requirem
ent
local_file = unpack_url(
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\operations\prepare.py", line 246, in unpack_url
_copy_source_tree(link.file_path, location)
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\site-packages\pip-20.2.4-py3.8.egg\pip\_internal\operations\prepare.py", line 186, in _copy_source_tree
shutil.copytree(source, target, **kwargs)
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\shutil.py", line 554, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "c:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.b842ddc51a36\python\install\lib\shutil.py", line 510, in _copytree
raise Error(errors)
shutil.Error: [('C:\\Users\\Matt\\hg\\.hg\\wcache\\checklink', 'C:\\Users\\Matt\\AppData\\Local\\Temp\\pip-req-build-w5r67snc\\.hg\\wcache\\checklink', "[Errno 22] Invalid argument: 'C:\\\\Users\\\\Matt\\\\hg\\\
\.hg\\\\wcache\\\\checklink'")]
Removed build tracker: 'C:\\Users\\Matt\\AppData\\Local\\Temp\\pip-req-tracker-n0i60uzg'
error[PIP_INSTALL_ERROR]: error running pip install: calling pip install
--> ./rust/hgcli\pyoxidizer.bzl:70:30
|
70 | exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pip_install()
error: error running pip install: calling pip install
make: *** [pyoxidizer] Error 1
make: Leaving directory `/c/Users/Matt/hg'
```
The fix is simply to remove the file on the WSL side, but it's not an immediate failure, so if there's a simple option to tell it to skip `.hg`, `.git`, etc when copying the tree, that would be helpful.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at rust/hgcli/pyoxidizer.bzl line 70, where exe.pip_install() is called, and trace the pip_install implementation that copies the source tree. Reproduce the failure with repository metadata such as .hg/wcache/checklink present. Done means pip_install avoids copying VCS metadata like .hg and .git while retaining the package source tree.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100