Bug: sam cli v1.135.0 and higher breaks sam local invoke in vscode
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 1.2k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 52
Description
### Description:
I have a SAM app that I debug locally. I haven't worked on SAM in awhile, so I decided to update the CLI from v1.133.0 to the latest v1.137.1. After doing so, I get the message `connect ECONNREFUSED 127.0.0.1:5858` in a VScode popup when debugging. I cancel the message, and it shows again about 6 times for me to click `cancel` - just to frustrate me. :)
As it was working fine before the SAM CLI upgrade, I downgraded until I found a version that works for local invoke/debug - 1.134.0. So, it appears something caused `sam local invoke` to break for me starting with v1.135.0.
I have a vscode launch.json file that I use with vscode `Start Debugging` to invoke sam local invoke
```
{
"version": "0.2.0",
"configurations": [
{
"type": "aws-sam",
"request": "direct-invoke",
"name": "TBOMAPIFunction-sandbox-thurber",
"invokeTarget": {
"target": "template",
"templatePath": "${workspaceFolder}/template.yaml",
"logicalId": "TBOMAPIFunction"
},
"lambda": {
"runtime": "python3.13",
"payload": {},
"memoryMb": 2048,
"environmentVariables": {
"redacted":"redacted"
}
},
"aws": {
"region": "us-west-2",
"credentials": "profile:Administrator-IP-Restricted-144REDACTED"
}
}
]
}
```
template.yaml follows. Note this lambda is scheduled with event bridge. There are no API gateway components
```yaml
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
TBOM API Interface
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Environment:
Variables:
POWERTOOLS_SERVICE_NAME: tbom
EnvironmentType: !Ref EnvironmentType
Timeout: 3
Tracing: Active
# You can add LoggingConfig parameters such as the Logformat, Log Group, and SystemLogLevel or ApplicationLogLevel.
# Learn more here https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-loggingconfig.
LoggingConfig:
LogFormat: JSON
Parameters:
EnvironmentType:
Type: String
ScheduleExpression:
Description: Schedule Expression used by Event Scheduler
Default: cron(0 5-17 ? * MON-FRI *)
Type: String
ScheduleTimezone:
Description: IANA time zone identifier used by Event Scheduler
Type: String
Default: "America/Indiana/Indianapolis"
SubnetIds:
Type: List
Description: Subnet IDs for the environment
SecurityGroupIds:
Type: List
Description: Security Group IDs for the environment
Resources:
TBOMSchedulerGroup:
Type: AWS::Scheduler::ScheduleGroup
Properties:
Name: TBOM
TBOMAPIFunction:
Type: AWS::Serverless::Function
DependsOn: TBOMSchedulerGroup
Properties:
FunctionUrlConfig:
AuthType: NONE
VpcConfig:
SecurityGroupIds: !Ref SecurityGroupIds
SubnetIds: !Ref SubnetIds
PackageType: Image
MemorySize: 2048
Timeout: 120
Architectures:
- x86_64
Policies:
- AWSLambdaVPCAccessExecutionRole
- SecretsManagerReadWrite
- Version: "2012-10-17" # Inline Policy Document
Statement:
- Effect: Allow
Action:
- "secretsmanager:BatchGetSecretValue"
- "secretsmanager:ListSecrets"
Resource: "*"
- AWSSecretsManagerGetSecretValuePolicy:
SecretArn: arn:aws:secretmanager:::redacted/path/goes/here/*
Events:
Call:
Type: ScheduleV2
Properties:
Description: Schedules the TBOM API lambda to be called periodically
GroupName: TBOM
FlexibleTimeWindow:
Mode: FLEXIBLE
MaximumWindowInMinutes: 10
RetryPolicy:
MaximumRetryAttempts: 5
MaximumEventAgeInSeconds: 300
ScheduleExpression:
Ref: ScheduleExpression
ScheduleExpressionTimezone:
Ref: ScheduleTimezone
Environment:
Variables:
REDACTED: Default value
Metadata:
Dockerfile: Dockerfile
DockerContext: ./tbom_api
DockerTag: python3.13-v1
```
This is the `AWS Toolkit Logs` Output (broken) on v1.35.0
```
2025-04-21 10:14:19.513 [info] Preparing to debug locally: Lambda "TBOMAPIFunction"
2025-04-21 10:14:19.513 [info] Building SAM application...
2025-04-21 10:14:19.581 [info] Command: (not started) [/usr/local/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctk0128cd42/output --template /home/ubuntu/dev/csinternal-tbom-api/csinternal-tbom-api/template.yaml]
2025-04-21 10:14:20.056 [warning] childProcess: Process 49762 exceeded cpu threshold: 100
2025-04-21 10:14:20.136 [info] Building codeuri: /home/ubuntu/dev/csinternal-tbom-api/csinternal-tbom-api runtime: None architecture: x86_64 functions: TBOMAPIFunction
2025-04-21 10:14:20.137 [info] Building image for TBOMAPIFunction function
2025-04-21 10:14:20.138 [info] Setting DockerBuildArgs for TBOMAPIFunction function
2025-04-21 10:14:20.155 [info] Step 1/8 : FROM public.ecr.aws/lambda/python:3.13
---> 839aea34b5c8
Step 2/8 : RUN pip install --upgrade pip==25.0.1 --no-cache-dir
---> Using cache
---> 5656344ca47d
2025-04-21 10:14:20.156 [info] Step 3/8 : COPY requirements.txt requirements-common.txt ./
---> Using cache
---> 1ac0767b408c
Step 4/8 : COPY *.py ./
---> Using cache
---> 33dade1de986
Step 5/8 : COPY Validation ${LAMBDA_TASK_ROOT}/Validation
---> Using cache
---> 2ecfc72c4e6d
Step 6/8 : ARG PIP_NO_CACHE_DIR=1
---> Using cache
---> 19cc29ee22c9
Step 7/8 : RUN python3.13 -m pip install --no-cache-dir -r requirements.txt --root-user-action=ignore --target .
---> Using cache
---> 578fa5420cca
Step 8/8 : CMD ["app.lambda_handler"]
---> Using cache
---> 70ac3b08a16e
[Warning] One or more build-args [SAM_BUILD_MODE] were not consumed
Successfully built 70ac3b08a16e
Successfully tagged tbomapifunction:python3.13-v1-debug
2025-04-21 10:14:20.159 [info] Build Succeeded
2025-04-21 10:14:20.159 [info] Built Artifacts : ../../../../../tmp/aws-toolkit-vscode/vsctk0128cd42/output
Built Template : ../../../../../tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke -t ../../../../../tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided --template-file ../../../../../tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml
2025-04-21 10:14:20.244 [info] Build complete.
2025-04-21 10:14:20.244 [info] Starting SAM application locally
2025-04-21 10:14:20.588 [info] AWS.running.command: Command: X {}
2025-04-21 10:14:20.588 [info] Command: (not started) [/usr/local/bin/sam local invoke TBOMAPIFunction --template /tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml --env-vars /tmp/aws-toolkit-vscode/vsctk0128cd42/env-vars.json -d 5858 --skip-pull-image --debugger-path /home/ubuntu/.vscode-server/extensions/amazonwebservices.aws-toolkit-vscode-3.55.0/resources/debugger --debug-args /var/lang/bin/python3.13 /tmp/lambci_debug_files/py_debug_wrapper.py --listen 0.0.0.0:5858 --wait-for-client --log-to-stderr /var/runtime/bootstrap.py --region us-west-2 --runtime python3.13]
2025-04-21 10:14:21.228 [info] Invoking Container created from tbomapifunction:python3.13-v1-debug
2025-04-21 10:14:21.239 [info] Requested to skip pulling images ...
2025-04-21 10:14:21.430 [info] START RequestId: 9030adce-ac29-40c6-91cf-b62447f16af2 Version: $LATEST
2025-04-21 10:14:21.445 [info] File "/var/lang/bin/python3.13", line 1
ELF
SyntaxError: source code cannot contain null bytes
2025-04-21 10:14:21.447 [info] 21 Apr 2025 10:14:21,447 [ERROR] (rapid) Init failed InvokeID= error=Runtime exited with error: exit status 1
2025-04-21 10:14:21.447 [info] 21 Apr 2025 10:14:21,447 [ERROR] (rapid) Invoke failed error=Runtime exited with error: exit status 1 InvokeID=3d55cf36-d4c7-4846-802f-964f29b75159
2025-04-21 10:14:21.448 [info] 21 Apr 2025 10:14:21,448 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2025-04-21 10:14:22.449 [info]
2025-04-21 10:14:22.807 [info] Attaching debugger to SAM application...
2025-04-21 10:21:12.431 [error] SamLaunchRequestError: Failed to attach debugger [DebuggerRetryLimit]
```
This is `AWS Toolkit Output` Logs on v1.134.0 (working)
```
2025-04-21 10:33:21.875 [info] Command: (not started) [/bin/bash -i -l -c '/home/ubuntu/.vscode-server/cli/servers/Stable-17baf841131aa23349f217ca7c570c76ee87b957/server/node' -p '"0925f1fcfc66" + JSON.stringify(process.env) + "0925f1fcfc66"']
2025-04-21 10:33:21.920 [warning] getUnixShellEnvironment#stderr:bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2025-04-21 10:33:22.820 [info] Preparing to debug locally: Lambda "TBOMAPIFunction"
2025-04-21 10:33:22.820 [info] Building SAM application...
2025-04-21 10:33:22.895 [info] Command: (not started) [/usr/local/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctk0c28d90e/output --template /home/ubuntu/dev/csinternal-tbom-api/csinternal-tbom-api/template.yaml]
2025-04-21 10:33:23.447 [info] Building codeuri: /home/ubuntu/dev/csinternal-tbom-api/csinternal-tbom-api runtime: None architecture: x86_64 functions: TBOMAPIFunction
2025-04-21 10:33:23.447 [info] Building image for TBOMAPIFunction function
2025-04-21 10:33:23.448 [info] Setting DockerBuildArgs for TBOMAPIFunction function
2025-04-21 10:33:23.468 [info] Step 1/8 : FROM public.ecr.aws/lambda/python:3.13
---> 839aea34b5c8
Step 2/8 : RUN pip install --upgrade pip==25.0.1 --no-cache-dir
2025-04-21 10:33:23.468 [info] ---> Using cache
---> 5656344ca47d
Step 3/8 : COPY requirements.txt requirements-common.txt ./
---> Using cache
---> 1ac0767b408c
Step 4/8 : COPY *.py ./
---> Using cache
---> 33dade1de986
Step 5/8 : COPY Validation ${LAMBDA_TASK_ROOT}/Validation
---> Using cache
---> 2ecfc72c4e6d
Step 6/8 : ARG PIP_NO_CACHE_DIR=1
---> Using cache
---> 19cc29ee22c9
Step 7/8 : RUN python3.13 -m pip install --no-cache-dir -r requirements.txt --root-user-action=ignore --target .
---> Using cache
---> 578fa5420cca
Step 8/8 : CMD ["app.lambda_handler"]
---> Using cache
---> 70ac3b08a16e
[Warning] One or more build-args [SAM_BUILD_MODE] were not consumed
Successfully built 70ac3b08a16e
Successfully tagged tbomapifunction:python3.13-v1-debug
2025-04-21 10:33:23.471 [info] Build Succeeded
2025-04-21 10:33:23.471 [info] Built Artifacts : ../../../../../tmp/aws-toolkit-vscode/vsctk0c28d90e/output
Built Template : ../../../../../tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke -t ../../../../../tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided --template-file ../../../../../tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml
2025-04-21 10:33:23.555 [info] Build complete.
2025-04-21 10:33:23.555 [info] Starting SAM application locally
2025-04-21 10:33:23.904 [info] AWS.running.command: Command: X {}
2025-04-21 10:33:23.904 [info] Command: (not started) [/usr/local/bin/sam local invoke TBOMAPIFunction --template /tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml --env-vars /tmp/aws-toolkit-vscode/vsctk0c28d90e/env-vars.json -d 5858 --skip-pull-image --debugger-path /home/ubuntu/.vscode-server/extensions/amazonwebservices.aws-toolkit-vscode-3.55.0/resources/debugger --debug-args /var/lang/bin/python3.13 /tmp/lambci_debug_files/py_debug_wrapper.py --listen 0.0.0.0:5858 --wait-for-client --log-to-stderr /var/runtime/bootstrap.py --region us-west-2]
2025-04-21 10:33:24.534 [info] Invoking Container created from tbomapifunction:python3.13-v1-debug
2025-04-21 10:33:24.542 [info] Building image...
2025-04-21 10:33:24.571 [info] .
2025-04-21 10:33:24.571 [info] ...........
2025-04-21 10:33:24.573 [info] .
2025-04-21 10:33:24.578 [info] .
2025-04-21 10:33:24.579 [info]
2025-04-21 10:33:24.585 [info] Requested to skip pulling images ...
2025-04-21 10:33:24.782 [info] START RequestId: bcb538e9-3ec0-4de6-93a6-03fc26166321 Version: $LATEST
2025-04-21 10:33:24.796 [info] /tmp/lambci_debug_files/py_debug_wrapper.py:22: SyntaxWarning: invalid escape sequence '\+'
if not re.search('(I|D)\+[0-9]+\.[0-9]+:', message) or self.debug:
2025-04-21 10:33:24.797 [info] Prepending Lambda task root to path: /var/task
Starting debugger...
2025-04-21 10:33:24.843 [info] 0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
2025-04-21 10:33:25.087 [info] Debugger waiting for client...
2025-04-21 10:33:25.146 [info] Attaching debugger to SAM application...
2025-04-21 10:33:40.815 [info] Debugger attached
```
From those logs, it looks like v1.135.0 is tossing in the parameter `--runtime python3.13` at the end of the sam local invoke.
```
#v1.135.0
/usr/local/bin/sam local invoke TBOMAPIFunction --template /tmp/aws-toolkit-vscode/vsctk0128cd42/output/template.yaml --env-vars /tmp/aws-toolkit-vscode/vsctk0128cd42/env-vars.json -d 5858 --skip-pull-image --debugger-path /home/ubuntu/.vscode-server/extensions/amazonwebservices.aws-toolkit-vscode-3.55.0/resources/debugger --debug-args /var/lang/bin/python3.13 /tmp/lambci_debug_files/py_debug_wrapper.py --listen 0.0.0.0:5858 --wait-for-client --log-to-stderr /var/runtime/bootstrap.py --region us-west-2 --runtime python3.13
#v1.134.0
/usr/local/bin/sam local invoke TBOMAPIFunction --template /tmp/aws-toolkit-vscode/vsctk0c28d90e/output/template.yaml --env-vars /tmp/aws-toolkit-vscode/vsctk0c28d90e/env-vars.json -d 5858 --skip-pull-image --debugger-path /home/ubuntu/.vscode-server/extensions/amazonwebservices.aws-toolkit-vscode-3.55.0/resources/debugger --debug-args /var/lang/bin/python3.13 /tmp/lambci_debug_files/py_debug_wrapper.py --listen 0.0.0.0:5858 --wait-for-client --log-to-stderr /var/runtime/bootstrap.py --region us-west-2
```
I wonder if this is causing the issue. Looking at the v135.0 release notes, this was implemented in [https://github.com/aws/aws-sam-cli/pull/7885](https://github.com/aws/aws-sam-cli/pull/7885)
Note those CLI commands (even for 1.134.0) noted above are not actually runnable from the command line. If I try to do so, I get an error `Error: No such option: --listen`
### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
1. OS: My desktop uses Windows, but I ssh to a remote Ubuntu 24 instance with the vscode `Connect to Host` feature.
2. `sam --version`: 1.134.0 and 1.135.0 and higher
3. AWS region: us-west-2
```
# Paste the output of `sam --info` here
```
```
{
"version": "1.134.0",
"system": {
"python": "3.11.10",
"os": "Linux-6.8.0-1027-aws-x86_64-with-glibc2.39"
},
"additional_dependencies": {
"docker_engine": "28.1.1",
"aws_cdk": "Not available",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
```
`Add --debug flag to command you are running`
I'm not sure how to do this since vscode is running the command for me.
Contributor guide
Assessment
This issue has not been assessed yet.