Intermittant TLS handshake failures in sam build with container build
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 1.2k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 52
Description
### Description:
Seeing intermittent failures when using the `--use-container` option of `sam build`. No changes in configuration caused this, it just started happening. SAM is being installed via https://github.com/aws-actions/setup-sam and I also attempted to pin the version to the latest one.
This appears to either be an issue with the way aws-sam-cli is doing a TLS handshake with ECR, or it is a failure with ECR.
### Steps to reproduce:
1. Use https://github.com/aws-actions/setup-sam actions
2. Use the `use-container` option.
### Observed result:
Docker image is sometimes not found due to 500 error / TLS error.
```
raise DockerImagePullFailedException(str(ex)) from ex
samcli.local.docker.manager.DockerImagePullFailedException: 500 Server Error: Internal Server Error ("Get "https://public.ecr.aws/v2/": net/http: TLS handshake timeout")
```
Full output, including build commands:
```bash
> npm run build && sam build --parallel --cached --use-container
> @0.0.1 build
> npm run clean && tsc
> @0.0.1 clean
> rm -fr build
Starting Build use cache
Starting Build inside a container
Cache is invalid, running build and copying resources to function build definition of
Fetching public.ecr.aws/sam/build-nodejs14.x:latest-x86_64 Docker container image.............................................................................................Cache is invalid, running build and copying resources to layer build definition of c23e0d67-4bb0-4ae6-ad5e-caffb11bda90
Building layer 'RuntimeDependenciesLayer'
For container layer build, first compatible runtime is chosen as build target for container.
.Cache is invalid, running build and copying resources to layer build definition of
Building layer ''
.For container layer build, first compatible runtime is chosen as build target for container.
.................................................................................................................................................................................................................................................................................................................................................
Mounting /home/runner/work//build as /tmp/samcli/source:ro,delegated inside runtime container
package.json file not found. Continuing the build without dependencies.
Running NodejsNpmBuilder:CopySource
Fetching public.ecr.aws/sam/build-nodejs14.x:latest-x86_64 Docker container image......
Mounting /home/runner/work/ as /tmp/samcli/source:ro,delegated inside runtime container
Running CustomMakeBuilder:CopySource
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : /tmp/samcli/artifacts
Traceback (most recent call last):
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.[10](https://github.com//runs/5880392319?check_suite_focus=true#step:7:10)/site-packages/docker/api/client.py", line 261, in _raise_for_status
response.raise_for_status()
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.35/images/create?tag=latest-x86_64&fromImage=public.ecr.aws%2Fsam%2Fbuild-nodejs14.x%3Alatest-x86_64
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 160, in pull_image
result_itr = self.docker_client.api.pull(image_name, tag=tag, stream=True, decode=True)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/docker/api/image.py", line 415, in pull
self._raise_for_status(response)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/docker/api/client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Get "https://public.ecr.aws/v2/": net/http: TLS handshake timeout")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 85, in create
self.pull_image(image_name)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 163, in pull_image
raise DockerImagePullFailedException(str(ex)) from ex
samcli.local.docker.manager.DockerImagePullFailedException: 500 Server Error: Internal Server Error ("Get "https://public.ecr.aws/v2/": net/http: TLS handshake timeout")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/runner/work/_temp/setup-sam-6sS74t/bin/sam", line 8, in
sys.exit(cli())
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/telemetry/metric.py", line 166, in wrapped
raise exception # pylint: disable=raising-bad-type
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/telemetry/metric.py", line 124, in wrapped
return_value = func(*args, **kwargs)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/utils/version_checker.py", line 41, in wrapped
actual_result = func(*args, **kwargs)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/cli/main.py", line 87, in wrapper
return func(*args, **kwargs)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/commands/build/command.py", line 174, in cli
do_cli(
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/commands/build/command.py", line 251, in do_cli
ctx.run()
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/commands/build/build_context.py", line 191, in run
build_result = builder.build()
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/app_builder.py", line 219, in build
return ApplicationBuildResult(build_graph, build_strategy.build())
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/build_strategy.py", line 364, in build
async_results = self._async_context.run_async()
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/utils/async_utils.py", line 131, in run_async
return run_given_tasks_async(self._async_tasks, event_loop)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/utils/async_utils.py", line 90, in run_given_tasks_async
return event_loop.run_until_complete(_run_given_tasks_async(tasks, event_loop, executor))
File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/utils/async_utils.py", line 58, in _run_given_tasks_async
raise result
File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/build_strategy.py", line 543, in build_single_layer_definition
return self._cached_build_strategy.build_single_layer_definition(layer_definition)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/build_strategy.py", line 304, in build_single_layer_definition
build_result = self._delegate_build_strategy.build_single_layer_definition(layer_definition)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/build_strategy.py", line 205, in build_single_layer_definition
layer.full_path: self._build_layer(
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/app_builder.py", line 524, in _build_layer
self._build_function_on_container(
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/lib/build/app_builder.py", line 816, in _build_function_on_container
self._container_manager.run(container)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line [11](https://github.com//runs/5880392319?check_suite_focus=true#step:7:11)5, in run
self.create(container)
File "/home/runner/work/_temp/setup-sam-6sS74t/.venv/lib/python3.10/site-packages/samcli/local/docker/manager.py", line 88, in create
raise DockerImagePullFailedException(
samcli.local.docker.manager.DockerImagePullFailedException: Could not find public.ecr.aws/sam/build-nodejs[14](https://github.com//runs/5880392319?check_suite_focus=true#step:7:14).x:latest-x86_[64](https://github.com//runs/5880392319?check_suite_focus=true#step:7:64) image locally and failed to pull it from docker.
Error: Process completed with exit code 1.
```
### Expected result:
The build should consistently not fail due to a Docker error.
### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
1. OS: Linux (Github Actions)
4. `sam --version`: any
5. AWS region: us-east-1 or us-east-2
Contributor guide
Assessment
This issue has not been assessed yet.