Bug: SAM does not write Layer build results to `build.toml` unless its table already exists
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 1.2k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 52
Description
### Description:
Starting with an empty environment (e.g. no `.aws-sam` directory or no `.aws-sam/build.toml` file), attempting to build a layer by itself:
```sh
sam build --cached --use-container MyLayer
```
does not result in the layer's build results being written to `.aws-sam/build.toml`. In fact, this file does not even get created.
This means that subsequent calls to the same build command will not use the cache, despite cached files being created in `.aws-sam/cache` and that a problem such as #5507 cannot be worked around.
**Edit:** currently, a Layer build will only _update_ its entry within `.aws-sam/build.toml` (e.g. SAM won't create this file just for building a Layer), essentially meaning that in order for a Layer to be cached, it must first at least have been attempted to be cached. A workaround for this situation is to force the creation of `.aws-sam/build.toml`, building in whatever way is necessary to generate the Layer's entry in this file (or otherwise manually creating the TOML would work too). In my case, it was quickest to just do:
```sh
sam build --cached MyFunction # this fails to build MyLayer because of issue #5507
# now, .aws-build/build.toml exists so we can carry on
sam build --cached --use-container MyLayer # this caches, updating build.toml
sam build --cached MyFunction # this succeeds now
```
### Steps to reproduce:
1. Remove any `.aws-sam` directory or `.aws-sam/build.toml` file
2. Define a Layer and Function such as this in `template.yaml`:
```yaml
MyLayer:
Type: AWS::Serverless::LayerVersion
Properties:
ContentUri: layers/mylayer/
CompatibleRuntimes:
- provided.al2
Metadata:
BuildMethod: makefile
MyFunction:
Type: AWS::Serverless::Function
Metadata:
BuildMethod: go1.x
Properties:
CodeUri: functions/myfunction
Handler: bootstrap
Runtime: provided.al2
Layers:
- !Ref MyLayer
```
```make
build-MyLayer:
mkdir "$(ARTIFACTS_DIR)/bin"
echo 'dummy' > "$(ARTIFACTS_DIR)/bin/dumy"
```
3. Build that layer with: `sam build --cached --use-container MyLayer`
4. Observe that no `.aws-sam/build.toml` file is created.
5. Try and re-run `sam build` command and layer will not have been cached.
### Observed result:
```sh
$ sam build --debug --cached --use-container MyLayer
2023-07-21 04:44:17,571 | No config file found in this directory.
2023-07-21 04:44:17,574 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/home/circleci/project2/samconfig.toml'
2023-07-21 04:44:17,575 | Config file location: /home/circleci/project2/samconfig.toml
2023-07-21 04:44:17,576 | Config file '/home/circleci/project2/samconfig.toml' does not exist
2023-07-21 04:44:17,578 | Using SAM Template at /home/circleci/project2/template.yml
2023-07-21 04:44:17,593 | Using config file: samconfig.toml, config environment: default
2023-07-21 04:44:17,594 | Expand command line arguments to:
2023-07-21 04:44:17,594 | --template_file=/home/circleci/project2/template.yml --cached --use_container --resource_logical_id=MyLayer --mount_with=READ --build_dir=.aws-sam/build
--cache_dir=.aws-sam/cache
2023-07-21 04:44:17,633 | 'build' command is called
2023-07-21 04:44:17,634 | Starting Build use cache
2023-07-21 04:44:17,635 | Starting Build inside a container
2023-07-21 04:44:17,637 | No Parameters detected in the template
2023-07-21 04:44:17,653 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:17,654 | 0 stacks found in the template
2023-07-21 04:44:17,654 | No Parameters detected in the template
2023-07-21 04:44:17,668 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:17,669 | 1 resources found in the stack
2023-07-21 04:44:17,669 | --base-dir is not presented, adjusting uri layers/mylayer/ relative to /home/circleci/project2/template.yml
2023-07-21 04:44:17,672 | 1 resources found in the stack
2023-07-21 04:44:17,674 | Instantiating build definitions
2023-07-21 04:44:17,674 | No previous build graph found, generating new one
2023-07-21 04:44:17,675 | Unique Layer build definition found, adding as new (Layer Build Definition: LayerBuildDefinition(MyLayer, /home/circleci/project2/layers/mylayer, ,
ef3c3c4f-ca7b-4153-9bbd-6e33b801fbd7, makefile, ['provided.al2'], x86_64, {}), Layer: )
2023-07-21 04:44:17,676 | Running cached build for runtime makefile for following resources (>)
2023-07-21 04:44:17,678 | Cache is invalid, running build and copying resources for following layers (MyLayer)
2023-07-21 04:44:17,679 | Building layer 'MyLayer'
2023-07-21 04:44:17,680 | For container layer build, first compatible runtime is chosen as build target for container.
Fetching public.ecr.aws/sam/build-provided.al2:latest-x86_64 Docker container image......
2023-07-21 04:44:17,788 | Mounting /home/circleci/project2/layers/mylayer as /tmp/samcli/source:ro,delegated, inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'RustCargoLambdaBuilder' with capability 'Capability(language='rust', dependency_manager='cargo', application_framework=None)'
Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Running workflow 'CustomMakeBuilder'
Running CustomMakeBuilder:CopySource
Copying source file (/tmp/samcli/source/Makefile) to destination (/tmp/samcli/scratch/Makefile)
CustomMakeBuilder:CopySource succeeded
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : /tmp/samcli/artifacts
executing Make: ['make', '--makefile', '/tmp/samcli/source/Makefile', 'build-MyLayer']
mkdir "/tmp/samcli/artifacts/bin"
echo 'dummy' > "/tmp/samcli/artifacts/bin/dumy"
CustomMakeBuilder:MakeBuild succeeded
2023-07-21 04:44:18,104 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "result": {"artifacts_dir": "/tmp/samcli/artifacts"}}
2023-07-21 04:44:18,105 | Build inside container was successful. Copying artifacts from container to host
2023-07-21 04:44:18,280 | Copying from container: /tmp/samcli/artifacts/. -> /home/circleci/project2/.aws-sam/build/MyLayer
2023-07-21 04:44:18,325 | Build inside container succeeded
2023-07-21 04:44:18,326 | Instantiating build definitions
2023-07-21 04:44:18,327 | No previous build graph found, generating new one
2023-07-21 04:44:18,328 | There is no customer defined id or cdk path defined for resource MyLayer, so we will use the resource logical id as the resource id
2023-07-21 04:44:18,329 | 1 resources found in the stack
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided
2023-07-21 04:44:18,330 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-07-21 04:44:18,332 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-07-21 04:44:18,332 | Unable to find Click Context for getting session_id.
$ ls .aws-sam/build.toml ls: .aws-sam/build.toml: No such file or directory
```
### Expected result:
Layer to be cached and recorded, available for subsequent runs of `sam build`.
### Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
```json
{
"version": "1.93.0",
"system": {
"python": "3.10.6",
"os": "Linux-5.19.0-1022-aws-x86_64-with-glibc2.35"
},
"additional_dependencies": {
"docker_engine": "20.10.24",
"aws_cdk": "Not available",
"terraform": "Not available"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
```
Contributor guide
Assessment
This issue has not been assessed yet.