aws / aws/aws-sam-cli

Bug: sam cli v1.135.0 and higher breaks sam local invoke in vscode

Open
#7,996 5 comments 0 reactions 0 assignees View on GitHub
stage/needs-triage
Dominant language
Python
Stars
6.7k
Forks
1.2k
Avg merge
1d 10h
Merged PRs (30d)
52

Description

### Description:
I have a SAM app that I debug locally. I haven't worked on SAM in awhile, so I decided to update the CLI from v1.133.0 to the latest v1.137.1. After doing so, I get the message `connect ECONNREFUSED 127.0.0.1:5858` in a VScode popup when debugging. I cancel the message, and it shows again about 6 times for me to click `cancel` - just to frustrate me. :)

As it was working fine before the SAM CLI upgrade, I downgraded until I found a version that works for local invoke/debug - 1.134.0. So, it appears something caused `sam local invoke` to break for me starting with v1.135.0.

I have a vscode launch.json file that I use with vscode `Start Debugging` to invoke sam local invoke

```
{
"version": "0.2.0",
"configurations": [
{
"type": "aws-sam",
"request": "direct-invoke",
"name": "TBOMAPIFunction-sandbox-thurber",
"invokeTarget": {
"target": "template",
"templatePath": "${workspaceFolder}/template.yaml",
"logicalId": "TBOMAPIFunction"
},
"lambda": {
"runtime": "python3.13",
"payload": {},
"memoryMb": 2048,
"environmentVariables": {
"redacted":"redacted"
}
},
"aws": {
"region": "us-west-2",
"credentials": "profile:Administrator-IP-Restricted-144REDACTED"
}
}
]
}
```

template.yaml follows. Note this lambda is scheduled with event bridge. There are no API gateway components

```yaml
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
TBOM API Interface

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Environment:
Variables:
POWERTOOLS_SERVICE_NAME: tbom
EnvironmentType: !Ref EnvironmentType
Timeout: 3
Tracing: Active

# You can add LoggingConfig parameters such as the Logformat, Log Group, and SystemLogLevel or ApplicationLogLevel.
# Learn more here https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-loggingconfig.
LoggingConfig:
LogFormat: JSON

Parameters:
EnvironmentType:
Type: String
ScheduleExpression:
Description: Schedule Expression used by Event Scheduler
Default: cron(0 5-17 ? * MON-FRI *)
Type: String
ScheduleTimezone:
Description: IANA time zone identifier used by Event Scheduler
Type: String
Default: "America/Indiana/Indianapolis"

SubnetIds:
Type: List
Description: Subnet IDs for the environment
SecurityGroupIds:
Type: List
Description: Security Group IDs for the environment

Resources:
TBOMSchedulerGroup:
Type: AWS::Scheduler::ScheduleGroup
Properties:
Name: TBOM

TBOMAPIFunction:
Type: AWS::Serverless::Function
DependsOn: TBOMSchedulerGroup
Properties:
FunctionUrlConfig:
AuthType: NONE
VpcConfig:
SecurityGroupIds: !Ref SecurityGroupIds
SubnetIds: !Ref SubnetIds
PackageType: Image
MemorySize: 2048
Timeout: 120
Architectures:
- x86_64
Policies:
- AWSLambdaVPCAccessExecutionRole
- SecretsManagerReadWrite
- Version: "2012-10-17" # Inline Policy Document
Statement:
- Effect: Allow
Action:
- "secretsmanager:BatchGetSecretValue"
- "secretsmanager:ListSecrets"
Resource: "*"
- AWSSecretsManagerGetSecretValuePolicy:
SecretArn: arn:aws:secretmanager:::redacted/path/goes/here/*
Events:
Call:
Type: ScheduleV2
Properties:
Description: Schedules the TBOM API lambda to be called periodically
GroupName: TBOM
FlexibleTimeWindow:
Mode: FLEXIBLE
MaximumWindowInMinutes: 10
RetryPolicy:
MaximumRetryAttempts: 5
MaximumEventAgeInSeconds: 300
ScheduleExpression:
Ref: ScheduleExpression
ScheduleExpressionTimezone:
Ref: ScheduleTimezone
Environment:
Variables:
REDACTED: Default value
Metadata:
Dockerfile: Dockerfile
DockerContext: ./tbom_api
DockerTag: python3.13-v1

```

This is the `AWS Toolkit Logs` Output (broken) on v1.35.0
```
2025-04-21 10:14:19.513 [info] Preparing to debug locally: Lambda "TBOMAPIFunction"
2025-04-21 10:14:19.513 [info] Building SAM application...
2025-04-21 10:14:19.581 [info] Command: (not started) [/usr/local/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctk0128cd42/output --template /home/ubuntu/dev/csinternal-tbom-api/csinternal-tbom-api/template.yaml]
2025-04-21 10:14:20.056 [warning] childProcess: Process 49762 exceeded cpu threshold: 100
2025-04-21 10:14:20.136 [info] Building codeuri: /home/ubuntu/dev/csinternal-tbom-api/csinternal-tbom-api runtime: None architecture: x86_64 functions: TBOMAPIFunction
2025-04-21 10:14:20.137 [info] Building image for TBOMAPIFunction function
2025-04-21 10:14:20.138 [info] Setting DockerBuildArgs for TBOMAPIFunction function
2025-04-21 10:14:20.155 [info] Step 1/8 : FROM public.ecr.aws/lambda/python:3.13
---> 839aea34b5c8
Step 2/8 : RUN pip install --upgrade pip==25.0.1 --no-cache-dir
---> Using cache
---> 5656344ca47d
2025-04-21 10:14:20.156 [info] Step 3/8 : COPY requirements.txt requirements-common.txt ./
---> Using cache
---> 1ac0767b408c
Step 4/8 : COPY *.py ./
---> Using cache
---> 33dade1de986
Step 5/8 : COPY Validation ${LAMBDA_TASK_ROOT}/Validation
---> Using cache
---> 2ecfc72c4e6d
Step 6/8 : ARG PIP_NO_CACHE_DIR=1
---> Using cache
---> 19cc29ee22c9
Step 7/8 : RUN python3.13 -m pip install --no-cache-dir -r requirements.txt --root-user-action=ignore --target .
---> Using cache
---> 578fa5420cca
Step 8/8 : CMD ["app.lambda_handler"]
---> Using cache
---> 70ac3b08a16e
[Warning] One or more build-args [SAM_BUILD_MODE] were not consumed
Successfully built 70ac3b08a16e
Successfully tagged tbomapifunction:python3.13-v1-debug
2025-04-21 10:14:20.159 [info] Build Succeeded
2025-04-21 10:14:20.159 [info] Built Artifacts : ../../../../../tmp/aws-toolkit-vscode/vsctk0128cd42/output
Built Template : ../../../../../tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke -t ../../../../../tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided --template-file ../../../../../tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml
2025-04-21 10:14:20.244 [info] Build complete.
2025-04-21 10:14:20.244 [info] Starting SAM application locally
2025-04-21 10:14:20.588 [info] AWS.running.command: Command: X {}
2025-04-21 10:14:20.588 [info] Command: (not started) [/usr/local/bin/sam local invoke TBOMAPIFunction --template /tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml --env-vars /tmp/aws-toolkit-vscode/vsctk0128cd42/env-vars.json -d 5858 --skip-pull-image --debugger-path /home/ubuntu/.vscode-server/extensions/amazonwebservices.aws-toolkit-vscode-3.55.0/resources/debugger --debug-args /var/lang/bin/python3.13 /tmp/lambci_debug_files/py_debug_wrapper.py --listen 0.0.0.0:5858 --wait-for-client --log-to-stderr /var/runtime/bootstrap.py --region us-west-2 --runtime python3.13]
2025-04-21 10:14:21.228 [info] Invoking Container created from tbomapifunction:python3.13-v1-debug
2025-04-21 10:14:21.239 [info] Requested to skip pulling images ...
2025-04-21 10:14:21.430 [info] START RequestId: 9030adce-ac29-40c6-91cf-b62447f16af2 Version: $LATEST
2025-04-21 10:14:21.445 [info] File "/var/lang/bin/python3.13", line 1
ELF
SyntaxError: source code cannot contain null bytes
2025-04-21 10:14:21.447 [info] 21 Apr 2025 10:14:21,447 [ERROR] (rapid) Init failed InvokeID= error=Runtime exited with error: exit status 1
2025-04-21 10:14:21.447 [info] 21 Apr 2025 10:14:21,447 [ERROR] (rapid) Invoke failed error=Runtime exited with error: exit status 1 InvokeID=3d55cf36-d4c7-4846-802f-964f29b75159
2025-04-21 10:14:21.448 [info] 21 Apr 2025 10:14:21,448 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2025-04-21 10:14:22.449 [info]
2025-04-21 10:14:22.807 [info] Attaching debugger to SAM application...
2025-04-21 10:21:12.431 [error] SamLaunchRequestError: Failed to attach debugger [DebuggerRetryLimit]
```

This is `AWS Toolkit Output` Logs on v1.134.0 (working)
```
2025-04-21 10:33:21.875 [info] Command: (not started) [/bin/bash -i -l -c '/home/ubuntu/.vscode-server/cli/servers/Stable-17baf841131aa23349f217ca7c570c76ee87b957/server/node' -p '"0925f1fcfc66" + JSON.stringify(process.env) + "0925f1fcfc66"']
2025-04-21 10:33:21.920 [warning] getUnixShellEnvironment#stderr:bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2025-04-21 10:33:22.820 [info] Preparing to debug locally: Lambda "TBOMAPIFunction"
2025-04-21 10:33:22.820 [info] Building SAM application...
2025-04-21 10:33:22.895 [info] Command: (not started) [/usr/local/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctk0c28d90e/output --template /home/ubuntu/dev/csinternal-tbom-api/csinternal-tbom-api/template.yaml]
2025-04-21 10:33:23.447 [info] Building codeuri: /home/ubuntu/dev/csinternal-tbom-api/csinternal-tbom-api runtime: None architecture: x86_64 functions: TBOMAPIFunction
2025-04-21 10:33:23.447 [info] Building image for TBOMAPIFunction function
2025-04-21 10:33:23.448 [info] Setting DockerBuildArgs for TBOMAPIFunction function
2025-04-21 10:33:23.468 [info] Step 1/8 : FROM public.ecr.aws/lambda/python:3.13
---> 839aea34b5c8
Step 2/8 : RUN pip install --upgrade pip==25.0.1 --no-cache-dir
2025-04-21 10:33:23.468 [info] ---> Using cache
---> 5656344ca47d
Step 3/8 : COPY requirements.txt requirements-common.txt ./
---> Using cache
---> 1ac0767b408c
Step 4/8 : COPY *.py ./
---> Using cache
---> 33dade1de986
Step 5/8 : COPY Validation ${LAMBDA_TASK_ROOT}/Validation
---> Using cache
---> 2ecfc72c4e6d
Step 6/8 : ARG PIP_NO_CACHE_DIR=1
---> Using cache
---> 19cc29ee22c9
Step 7/8 : RUN python3.13 -m pip install --no-cache-dir -r requirements.txt --root-user-action=ignore --target .
---> Using cache
---> 578fa5420cca
Step 8/8 : CMD ["app.lambda_handler"]
---> Using cache
---> 70ac3b08a16e
[Warning] One or more build-args [SAM_BUILD_MODE] were not consumed
Successfully built 70ac3b08a16e
Successfully tagged tbomapifunction:python3.13-v1-debug
2025-04-21 10:33:23.471 [info] Build Succeeded
2025-04-21 10:33:23.471 [info] Built Artifacts : ../../../../../tmp/aws-toolkit-vscode/vsctk0c28d90e/output
Built Template : ../../../../../tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke -t ../../../../../tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided --template-file ../../../../../tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml
2025-04-21 10:33:23.555 [info] Build complete.
2025-04-21 10:33:23.555 [info] Starting SAM application locally
2025-04-21 10:33:23.904 [info] AWS.running.command: Command: X {}
2025-04-21 10:33:23.904 [info] Command: (not started) [/usr/local/bin/sam local invoke TBOMAPIFunction --template /tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml --env-vars /tmp/aws-toolkit-vscode/vsctk0c28d90e/env-vars.json -d 5858 --skip-pull-image --debugger-path /home/ubuntu/.vscode-server/extensions/amazonwebservices.aws-toolkit-vscode-3.55.0/resources/debugger --debug-args /var/lang/bin/python3.13 /tmp/lambci_debug_files/py_debug_wrapper.py --listen 0.0.0.0:5858 --wait-for-client --log-to-stderr /var/runtime/bootstrap.py --region us-west-2]
2025-04-21 10:33:24.534 [info] Invoking Container created from tbomapifunction:python3.13-v1-debug
2025-04-21 10:33:24.542 [info] Building image...
2025-04-21 10:33:24.571 [info] .
2025-04-21 10:33:24.571 [info] ...........
2025-04-21 10:33:24.573 [info] .
2025-04-21 10:33:24.578 [info] .
2025-04-21 10:33:24.579 [info]
2025-04-21 10:33:24.585 [info] Requested to skip pulling images ...
2025-04-21 10:33:24.782 [info] START RequestId: bcb538e9-3ec0-4de6-93a6-03fc26166321 Version: $LATEST
2025-04-21 10:33:24.796 [info] /tmp/lambci_debug_files/py_debug_wrapper.py:22: SyntaxWarning: invalid escape sequence '\+'
if not re.search('(I|D)\+[0-9]+\.[0-9]+:', message) or self.debug:
2025-04-21 10:33:24.797 [info] Prepending Lambda task root to path: /var/task
Starting debugger...
2025-04-21 10:33:24.843 [info] 0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
2025-04-21 10:33:25.087 [info] Debugger waiting for client...
2025-04-21 10:33:25.146 [info] Attaching debugger to SAM application...
2025-04-21 10:33:40.815 [info] Debugger attached
```

From those logs, it looks like v1.135.0 is tossing in the parameter `--runtime python3.13` at the end of the sam local invoke.

```
#v1.135.0
/usr/local/bin/sam local invoke TBOMAPIFunction --template /tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml --env-vars /tmp/aws-toolkit-vscode/vsctk0128cd42/env-vars.json -d 5858 --skip-pull-image --debugger-path /home/ubuntu/.vscode-server/extensions/amazonwebservices.aws-toolkit-vscode-3.55.0/resources/debugger --debug-args /var/lang/bin/python3.13 /tmp/lambci_debug_files/py_debug_wrapper.py --listen 0.0.0.0:5858 --wait-for-client --log-to-stderr /var/runtime/bootstrap.py --region us-west-2 --runtime python3.13

#v1.134.0
/usr/local/bin/sam local invoke TBOMAPIFunction --template /tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml --env-vars /tmp/aws-toolkit-vscode/vsctk0c28d90e/env-vars.json -d 5858 --skip-pull-image --debugger-path /home/ubuntu/.vscode-server/extensions/amazonwebservices.aws-toolkit-vscode-3.55.0/resources/debugger --debug-args /var/lang/bin/python3.13 /tmp/lambci_debug_files/py_debug_wrapper.py --listen 0.0.0.0:5858 --wait-for-client --log-to-stderr /var/runtime/bootstrap.py --region us-west-2

```

I wonder if this is causing the issue. Looking at the v135.0 release notes, this was implemented in [https://github.com/aws/aws-sam-cli/pull/7885](https://github.com/aws/aws-sam-cli/pull/7885)

Note those CLI commands (even for 1.134.0) noted above are not actually runnable from the command line. If I try to do so, I get an error `Error: No such option: --listen`

### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: My desktop uses Windows, but I ssh to a remote Ubuntu 24 instance with the vscode `Connect to Host` feature.
2. `sam --version`: 1.134.0 and 1.135.0 and higher
3. AWS region: us-west-2

```
# Paste the output of `sam --info` here
```
```
{
"version": "1.134.0",
"system": {
"python": "3.11.10",
"os": "Linux-6.8.0-1027-aws-x86_64-with-glibc2.39"
},
"additional_dependencies": {
"docker_engine": "28.1.1",
"aws_cdk": "Not available",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
```

`Add --debug flag to command you are running`

I'm not sure how to do this since vscode is running the command for me.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.