git dependencies in pip don't install on Windows
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 1.2k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 52
Description
### Description
I'm not able to use git/ssh based dependencies using PIP and SAM.
Using a git dependency in my requirements.txt file like this:
git+ssh://git@bitbucket.org/myaccount/example.git
When running sam build from a Windows command prompt, it gives an error like thi:
AssertionError: name 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-puwxrmxy\\.git' doesn't start with prefix 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-puwxrmxy\\'
### Steps to reproduce
From a Windows 10 command prompt:
1. sam init -r python3.6 -n git-dependencies
2. Edit requirements.txt and add in a git dependency:
git+ssh://git@bitbucket.org/myaccount/somethingorother.git
3. sam build
### Observed result
c:\temp\git-dependencies>sam build --debug
2019-04-23 13:51:15 Using SAM Template at c:\temp\git-dependencies\template.yaml
2019-04-23 13:51:15 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-04-23 13:51:15 Changing event name from before-call.apigateway to before-call.api-gateway
2019-04-23 13:51:15 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-04-23 13:51:15 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-04-23 13:51:15 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-04-23 13:51:15 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-04-23 13:51:15 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-04-23 13:51:15 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-04-23 13:51:15 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-04-23 13:51:15 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-04-23 13:51:15 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-04-23 13:51:15 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-04-23 13:51:15 Changing event name from before-call.apigateway to before-call.api-gateway
2019-04-23 13:51:15 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-04-23 13:51:15 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-04-23 13:51:15 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-04-23 13:51:15 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-04-23 13:51:15 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-04-23 13:51:15 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-04-23 13:51:15 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-04-23 13:51:15 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-04-23 13:51:15 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-04-23 13:51:15 'build' command is called
2019-04-23 13:51:15 No Parameters detected in the template
2019-04-23 13:51:15 2 resources found in the template
2019-04-23 13:51:15 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2019-04-23 13:51:15 Building resource 'HelloWorldFunction'
2019-04-23 13:51:15 Loading workflow module 'aws_lambda_builders.workflows'
2019-04-23 13:51:15 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2019-04-23 13:51:15 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-04-23 13:51:15 Running workflow 'PythonPipBuilder'
2019-04-23 13:51:15 Running PythonPipBuilder:ResolveDependencies
2019-04-23 13:51:15 calling pip download -r C:\temp\git-dependencies\hello_world\requirements.txt --dest C:\Users\johnc\AppData\Local\Temp\tmpjm5d5o79
2019-04-23 13:51:17 PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 39, in execute
self.scratch_dir
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 143, in build_dependencies
requirements_path, artifacts_dir_path, scratch_dir_path)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 209, in build_site_packages
scratch_directory, requirements_filepath)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 234, in _download_dependencies
requirements_filename, directory)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 330, in _download_all_dependencies
self._pip.download_all_dependencies(requirements_filename, directory)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 632, in download_all_dependencies
raise PackageDownloadError(error)
aws_lambda_builders.workflows.python_pip.packager.PackageDownloadError: Exception:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
status = self.run(options, args)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\commands\download.py", line 164, in run
resolver.resolve(requirement_set)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\operations\prepare.py", line 353, in prepare_linked_requirement
req.archive(self.download_dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 873, in archive
rootdir=dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 831, in _get_archive_name
name = self._clean_zip_name(path, rootdir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 822, in _clean_zip_name
"name %r doesn't start with prefix %r" % (name, prefix)
AssertionError: name 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\.git' doesn't start with prefix 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflow.py", line 248, in run
action.execute()
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 42, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Exception:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
status = self.run(options, args)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\commands\download.py", line 164, in run
resolver.resolve(requirement_set)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\operations\prepare.py", line 353, in prepare_linked_requirement
req.archive(self.download_dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 873, in archive
rootdir=dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 831, in _get_archive_name
name = self._clean_zip_name(path, rootdir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 822, in _clean_zip_name
"name %r doesn't start with prefix %r" % (name, prefix)
AssertionError: name 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\.git' doesn't start with prefix 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\'
Build Failed
Error: PythonPipBuilder:ResolveDependencies - Exception:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
status = self.run(options, args)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\commands\download.py", line 164, in run
resolver.resolve(requirement_set)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\operations\prepare.py", line 353, in prepare_linked_requirement
req.archive(self.download_dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 873, in archive
rootdir=dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 831, in _get_archive_name
name = self._clean_zip_name(path, rootdir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 822, in _clean_zip_name
"name %r doesn't start with prefix %r" % (name, prefix)
AssertionError: name 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\.git' doesn't start with prefix 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\'
### Expected result
Should have installed dependency from git.
### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
1. OS: Windows 10
2. `sam --version`:0.15.0
3. python version: 3.6.7
4. pip version: 19.0.3
If I run from Ubuntu WSL then it installs OK. (But not if I use "--use-container", but then that never works for me from WSL anyway).
It also doesn't work from Windows if I add "--use-container" in which case I get:
Error: PythonPipBuilder:ResolveDependencies - Command "git clone -q ssh://git@bitbucket.org/myaccount/sample.git /tmp/pip-req-build-1n5kwnr1" failed with error code 128 in None
Is doing this sort of thing supported with "--use-container"? (If not, I can maybe raise it as a feature request)
From Windows, installing using pip directly works, i.e. like this:
pip install -r src\requirements.txt -t .aws-sam\tests
So I know that the requirements file is good, and that Windows, git and pip are playing nicely together.
Contributor guide
Assessment
This issue has not been assessed yet.