aws / aws/aws-sam-cli

Bug: sam local invoke unconditionally rebuilds docker image on every invocation when a layer is declared

Open
#9,255 3 comments 0 reactions 0 assignees View on GitHub
area/build need-customer-response
Dominant language
Python
Stars
6.7k
Forks
1.2k
Avg merge
1d 10h
Merged PRs (30d)
52

Description

### Description:
`sam local invoke` unconditionally rebuilds the Docker image on every single invocation when the function has a declared layer in the template (`AWS::Serverless::LayerVersion` with a local `ContentUri`).

Root cause (traced in source, v1.165.0):

- `samcli/local/docker/lambda_image.py:276-281` forces `_build_image()` whenever `any(layer.is_defined_within_template for layer in downloaded_layers)` is true — no hash/content comparison, this is a blanket condition.
- `samcli/lib/providers/provider.py:301` sets `is_defined_within_template = bool(codeuri)` — true for any layer built locally, unconditionally, regardless of whether its content actually changed since the last build/invoke.
- The rebuild itself (`_build_image` → `create_tarball`, `samcli/lib/utils/tar.py:51-53`) re-tars the entire layer directory via Python's `tarfile`, one file at a time, before Docker's build API/cache is ever reached. On our layer (~30MB, ~5300 files under `node_modules`) this alone costs 60-90s, every call.

SAM CLI already has the infrastructure to avoid this: `sam build` itself decides whether to reinstall a layer's dependencies by hashing the manifest (`package.json`/lockfile) via `DependencyHashGenerator` in `samcli/lib/build/build_strategy.py`, not by hashing/re-reading the installed `node_modules` tree. `lambda_image.py`'s invoke-time rebuild check does not reuse this at all.

### Steps to reproduce:
1. Template with a function that has `Layers: [!Ref DepsLayer]`, where `DepsLayer` is `AWS::Serverless::LayerVersion` with a local `ContentUri` and `Metadata: BuildMethod: makefile` (i.e. built locally, not referencing a published/remote layer ARN).
2. `sam build`
3. `sam local invoke MyFunction -e event.json --debug`
4. Immediately run the exact same command again, with no changes to any file.
5. Both invocations pay the same "Building image" cost.

### Observed result:
Two consecutive invokes, same template, same build, zero changes in between:

=== invoke 1 ===
2026-09-04 19:31:57,863 | Local image is up-to-date
2026-09-04 19:33:24,098 | Mounting /path/.aws-sam/build/MyLambda as /var/task:ro,delegated, inside runtime container
sam local invoke MyLambda -e event.json --debug 1:31.82 total

=== invoke 2 ===
2026-09-04 19:33:29,058 | Local image is up-to-date
2026-09-04 19:34:29,438 | Mounting /path/.aws-sam/build/MyLambda as /var/task:ro,delegated, inside runtime container
sam local invoke MyLambda -e event.json --debug 1:05.05 total

86s and 60s respectively spent between "Local image is up-to-date" and "Mounting" — the silent image-rebuild window — on both calls, despite nothing having changed between them.

### Expected result:
`sam local invoke` should reuse the existing rapid image as much as possible — the same way SAM already skips reinstalling a layer's dependencies during `sam build` when its manifest hash is unchanged.

### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 26.6.2 (arm64)
2. `sam --version`: SAM CLI, version 1.165.0

{
"version": "1.165.0",
"system": {
"python": "3.14.7",
"os": "macOS-26.6.2-arm64-arm-64bit-Mach-O"
},
"additional_dependencies": {
"container_engine": "Docker(v29.7.2)",
"aws_cdk": "Not available",
"terraform": "1.7.4"
}
}

Contributor guide

Open the contributing guide

Research direction

Start in samcli/local/docker/lambda_image.py around lines 276-281 and samcli/lib/providers/provider.py around line 301 to trace why locally defined layers force _build_image(). Read samcli/lib/build/build_strategy.py and samcli/lib/utils/tar.py to compare existing manifest-hash behavior with invoke-time tarball creation. Done means unchanged consecutive sam local invoke runs no longer rebuild the layer image, while changed layer inputs still trigger a rebuild.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
cli, devops, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.