aws / aws/aws-sam-cli

Bug: sam local start-api not working in a docker-out-of-docker setup, due to an empty /var/task directory.

Open
#7,029 4 comments 0 reactions 0 assignees View on GitHub
area/docker area/local type/feature
Dominant language
Python
Stars
6.7k
Forks
1.2k
Avg merge
1d 10h
Merged PRs (30d)
52

Description

### Description:

When running `sam local start-api` in a docker-out-of-docker setup, it seems that the `/var/task` folder mounted into the lambda-executing container is not populated with the zip containing the lambda binary.

### Steps to reproduce:

1. Create a simple JS hello-world, zip it and drop to a common path - here `/tmp/lambda-bins`

2. Create a basic setup with an API Gateway and one lambda endpoint:
```
Resources:
HelloWorld:
Type: AWS::Serverless::Function
Properties:
FunctionName: HelloWorld
Handler: index.handler
Runtime: nodejs16.x
CodeUri: ./lambda-hello-world.zip

ApiGateway:
Type: AWS::Serverless::Api
Properties:
Name: api-gateway
StageName: prod
DefinitionBody:
swagger: '2.0'
paths:
/hello:
get:
produces:
- application/json
responses:
200:
description: '200 response'
x-amazon-apigateway-integration:
httpMethod: GET
type: aws_proxy
uri:
Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorld.Arn}/invocations
```

3. Create a simple docker container with the aws-sam-cli.
```dockerfile
FROM python:3.10-alpine

RUN apk add --no-cache --virtual build-deps build-base libffi-dev gcc \
&& pip install --no-cache-dir aws-sam-cli==1.116.0 \
&& pip3 uninstall --yes pip \
&& apk del build-deps
```
4. Create a compose file to spin everything up:
```
networks:
sam_subnet:
driver: bridge
name: test-sub

services:
sam-lambda:
build: .
working_dir: /opt/data
ports:
- 3000:3000
volumes:
# Docker
- /var/run/docker.sock:/var/run/docker.sock
# SAM template
- ./template.yaml:/opt/data/template.yaml
# Directory on the host with the lambda binaries
- /tmp/lambda-bins:/tmp/lambda-bins
networks:
- sam_subnet
command:
[
"sam",
"local",
"start-api",
"--debug",
"--template",
"template.yaml",
"--host",
"0.0.0.0",
"--docker-network",
"test-sub",
"--docker-volume-basedir",
"/tmp/lambda-bins",
"--container-host-interface",
"0.0.0.0",
"--container-host",
"host.docker.internal",
]
extra_hosts:
- "host.docker.internal:host-gateway"
```
A few points of interest:
- the working directory for the aws-sam-cli container is `/opt/data`. It will only contain the sam template.
- the lambda binaries are shared via another mounted folder `/tmp/lambda-bins`

5. Call the lambda function using curl:
```
curl -v http://localhost:3000/hello
```

### Observed result:

The lambda function fails to execute with the following:
```
sam-lambda-1 | 2024-05-06 17:41:09,702 | Found one Lambda function with name 'HelloWorld'
sam-lambda-1 | 2024-05-06 17:41:09,702 | Invoking index.handler (nodejs16.x)
sam-lambda-1 | 2024-05-06 17:41:09,702 | No environment variables found for function 'HelloWorld'
sam-lambda-1 | 2024-05-06 17:41:09,702 | Loading AWS credentials from session with profile 'None'
sam-lambda-1 | 2024-05-06 17:41:11,714 | Resolving code path. Cwd=/tmp/lambda-bins, CodeUri=/tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1 | 2024-05-06 17:41:11,714 | Resolved absolute path to code is /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1 | 2024-05-06 17:41:11,714 | Resolving code path. Cwd=/opt/data, CodeUri=/tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1 | 2024-05-06 17:41:11,714 | Resolved real code path to /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1 | 2024-05-06 17:41:11,715 | Decompressing /tmp/lambda-bins/lambda-hello-world.zip
sam-lambda-1 | 2024-05-06 17:41:12,404 | Local image is up-to-date
sam-lambda-1 | 2024-05-06 17:41:12,409 | Checking free port on 0.0.0.0:5345
sam-lambda-1 | 2024-05-06 17:41:12,411 | Using local image: public.ecr.aws/lambda/nodejs:16-rapid-x86_64.
sam-lambda-1 |
sam-lambda-1 | 2024-05-06 17:41:12,411 | Mounting /tmp/tmpqn5dh6cm as /var/task:ro,delegated, inside runtime container
sam-lambda-1 | 2024-05-06 17:41:12,833 | Starting a timer for 3 seconds for function 'HelloWorld'
sam-lambda-1 | 2024-05-06 17:41:12,833 | Getting lock for the key host.docker.internal-5345
sam-lambda-1 | 2024-05-06 17:41:12,833 | Waiting to retrieve the lock (host.docker.internal-5345) to start invocation
sam-lambda-1 | START RequestId: 37f0705f-d4e9-4a29-922f-60307d74dc08 Version: $LATEST
sam-lambda-1 | 2024-05-06T17:41:12.941Z undefined ERROR Uncaught Exception {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'index'\nRequire stack:\n- /var/runtime/index.mjs","stack":["Runtime.ImportModuleError: Error: Cannot find module 'index'","Require stack:","- /var/runtime/index.mjs"," at _loadUserApp (file:///var/runtime/index.mjs:1087:17)"," at async Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)"," at async start (file:///var/runtime/index.mjs:1282:23)"," at async file:///var/runtime/index.mjs:1288:1"]}

```
As one can see, although the binary seems properly located as per this line:
`sam-lambda-1 | 2024-05-06 17:41:11,715 | Decompressing /tmp/lambda-bins/lambda-hello-world.zip`

It seems that `/var/task` ends up being empty in the lambda-executing container.

### Expected result:

We would expect the binary to be mounted into the lambda-executing container, and then the lambda function to properly execute.

### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux Ubuntu 24.04
2. `sam --version`: 1.116.0
3. AWS region: None

```
# Paste the output of `sam --info` here
sam-lambda-1 | {
sam-lambda-1 | "version": "1.116.0",
sam-lambda-1 | "system": {
sam-lambda-1 | "python": "3.10.14",
sam-lambda-1 | "os": "Linux-6.8.0-31-generic-x86_64-with"
sam-lambda-1 | },
sam-lambda-1 | "additional_dependencies": {
sam-lambda-1 | "docker_engine": "26.1.1",
sam-lambda-1 | "aws_cdk": "Not available",
sam-lambda-1 | "terraform": "Not available"
sam-lambda-1 | },
sam-lambda-1 | "available_beta_feature_env_vars": [
sam-lambda-1 | "SAM_CLI_BETA_FEATURES",
sam-lambda-1 | "SAM_CLI_BETA_BUILD_PERFORMANCE",
sam-lambda-1 | "SAM_CLI_BETA_TERRAFORM_SUPPORT",
sam-lambda-1 | "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
sam-lambda-1 | ]
sam-lambda-1 | }
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.