aws / aws/aws-sam-cli

git dependencies in pip don't install on Windows

Open
#1,144 4 comments 0 reactions 0 assignees View on GitHub
area/build maintainer/need-followup
Dominant language
Python
Stars
6.7k
Forks
1.2k
Avg merge
1d 10h
Merged PRs (30d)
52

Description

### Description

I'm not able to use git/ssh based dependencies using PIP and SAM.

Using a git dependency in my requirements.txt file like this:
git+ssh://git@bitbucket.org/myaccount/example.git

When running sam build from a Windows command prompt, it gives an error like thi:
AssertionError: name 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-puwxrmxy\\.git' doesn't start with prefix 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-puwxrmxy\\'

### Steps to reproduce

From a Windows 10 command prompt:

1. sam init -r python3.6 -n git-dependencies
2. Edit requirements.txt and add in a git dependency:
git+ssh://git@bitbucket.org/myaccount/somethingorother.git
3. sam build

### Observed result

c:\temp\git-dependencies>sam build --debug
2019-04-23 13:51:15 Using SAM Template at c:\temp\git-dependencies\template.yaml
2019-04-23 13:51:15 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-04-23 13:51:15 Changing event name from before-call.apigateway to before-call.api-gateway
2019-04-23 13:51:15 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-04-23 13:51:15 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-04-23 13:51:15 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-04-23 13:51:15 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-04-23 13:51:15 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-04-23 13:51:15 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-04-23 13:51:15 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-04-23 13:51:15 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-04-23 13:51:15 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-04-23 13:51:15 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-04-23 13:51:15 Changing event name from before-call.apigateway to before-call.api-gateway
2019-04-23 13:51:15 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-04-23 13:51:15 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-04-23 13:51:15 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-04-23 13:51:15 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-04-23 13:51:15 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-04-23 13:51:15 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-04-23 13:51:15 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-04-23 13:51:15 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-04-23 13:51:15 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-04-23 13:51:15 'build' command is called
2019-04-23 13:51:15 No Parameters detected in the template
2019-04-23 13:51:15 2 resources found in the template
2019-04-23 13:51:15 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2019-04-23 13:51:15 Building resource 'HelloWorldFunction'
2019-04-23 13:51:15 Loading workflow module 'aws_lambda_builders.workflows'
2019-04-23 13:51:15 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2019-04-23 13:51:15 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2019-04-23 13:51:15 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2019-04-23 13:51:15 Running workflow 'PythonPipBuilder'
2019-04-23 13:51:15 Running PythonPipBuilder:ResolveDependencies
2019-04-23 13:51:15 calling pip download -r C:\temp\git-dependencies\hello_world\requirements.txt --dest C:\Users\johnc\AppData\Local\Temp\tmpjm5d5o79
2019-04-23 13:51:17 PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 39, in execute
self.scratch_dir
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 143, in build_dependencies
requirements_path, artifacts_dir_path, scratch_dir_path)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 209, in build_site_packages
scratch_directory, requirements_filepath)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 234, in _download_dependencies
requirements_filename, directory)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 330, in _download_all_dependencies
self._pip.download_all_dependencies(requirements_filename, directory)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 632, in download_all_dependencies
raise PackageDownloadError(error)
aws_lambda_builders.workflows.python_pip.packager.PackageDownloadError: Exception:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
status = self.run(options, args)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\commands\download.py", line 164, in run
resolver.resolve(requirement_set)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\operations\prepare.py", line 353, in prepare_linked_requirement
req.archive(self.download_dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 873, in archive
rootdir=dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 831, in _get_archive_name
name = self._clean_zip_name(path, rootdir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 822, in _clean_zip_name
"name %r doesn't start with prefix %r" % (name, prefix)
AssertionError: name 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\.git' doesn't start with prefix 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflow.py", line 248, in run
action.execute()
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 42, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Exception:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
status = self.run(options, args)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\commands\download.py", line 164, in run
resolver.resolve(requirement_set)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\operations\prepare.py", line 353, in prepare_linked_requirement
req.archive(self.download_dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 873, in archive
rootdir=dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 831, in _get_archive_name
name = self._clean_zip_name(path, rootdir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 822, in _clean_zip_name
"name %r doesn't start with prefix %r" % (name, prefix)
AssertionError: name 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\.git' doesn't start with prefix 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\'

Build Failed
Error: PythonPipBuilder:ResolveDependencies - Exception:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
status = self.run(options, args)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\commands\download.py", line 164, in run
resolver.resolve(requirement_set)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\operations\prepare.py", line 353, in prepare_linked_requirement
req.archive(self.download_dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 873, in archive
rootdir=dir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 831, in _get_archive_name
name = self._clean_zip_name(path, rootdir)
File "C:\Program Files\Python36\lib\site-packages\pip\_internal\req\req_install.py", line 822, in _clean_zip_name
"name %r doesn't start with prefix %r" % (name, prefix)
AssertionError: name 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\.git' doesn't start with prefix 'c:\\users\\johnc\\appdata\\local\\temp\\pip-req-build-unr3otwk\\'

### Expected result

Should have installed dependency from git.

### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10
2. `sam --version`:0.15.0
3. python version: 3.6.7
4. pip version: 19.0.3

If I run from Ubuntu WSL then it installs OK. (But not if I use "--use-container", but then that never works for me from WSL anyway).

It also doesn't work from Windows if I add "--use-container" in which case I get:

Error: PythonPipBuilder:ResolveDependencies - Command "git clone -q ssh://git@bitbucket.org/myaccount/sample.git /tmp/pip-req-build-1n5kwnr1" failed with error code 128 in None

Is doing this sort of thing supported with "--use-container"? (If not, I can maybe raise it as a feature request)

From Windows, installing using pip directly works, i.e. like this:
pip install -r src\requirements.txt -t .aws-sam\tests

So I know that the requirements file is good, and that Windows, git and pip are playing nicely together.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.