aws / aws/aws-sam-cli

Feature request: option for container-based builds for specific resources

Open
#5,507 3 comments 0 reactions 0 assignees View on GitHub
area/build type/feature
Dominant language
Python
Stars
6.7k
Forks
1.2k
Avg merge
1d 10h
Merged PRs (30d)
52

Description

### Describe your idea/feature/enhancement

Building resources - `LayerVersion`s and `Function`s - can currently occur either on the host natively or with `sam build --use-container` to build within a Docker container, but this an an all-or-nothing option - either everything builds in a container, or nothing does.

This causes an issue when a resource consists of or depends on native code, which necessitates that it _must_ be built in a container (matching the runtime environment), and attempting to use it together with other resources that needs to be run on the host or otherwise do not work with container builds (such as [`go1.x` on `provided.al2`/`provided.al2023`](https://github.com/aws/aws-sam-cli/issues/5280)). In real terms, this problem occurs with a Go `Function` attempting to depend on a native-code `LayerVersion` - the Function must be built on the host but the LayerVersion can only be built correctly in an container matching the Amazon Linux runtime. Similarly, the same issue occurs when intermixing multiple `Function`s in a configuration, such as Node.js, where some functions rely on native modules (requiring container-based builds) and others which do not.

If SAM supported the ability to explicitly set how a given resource should be to be built, this would resolve the issue and allow a single call to `sam build` to build all resources correctly.

### Proposal

Propose adding another `Metadata` resource attribute to the SAM specification like so:

```yaml
Metadata:
BuildMethod: makefile
BuildArchitecture: arm64
BuildType: [host|container]
```

or alternatively, implementing a CLI flag and adding support to `samconfig.toml`, similar to `build_image`, where individual resources can be set to explicitly use or not use container builds.

### Additional Details

It is possible to work around this issue by manually building each Layer and Function individually with the correct settings (e.g. running `sam build --use-container ResourceName` for parts that need it and then without `--use-container` for other resources), but this relies on the user mentally construct the dependency chains and effectively managing the caching performed by SAM. If one forgets to fully build _all_ the layers required by a function with the correct settings or if SAM considers one cache element out of date, then the build will produce incompatible resources and fail to work.

Additionally, there are challenges when mixing container-based and host-based builds since build processes may differ between build types. For example, with `nodejs20.x` runtimes, a container-based build will store cached artefacts into `.aws-sam/cache/{uuid}` but a host-based build looks at `.aws-sam/deps/{uuid}` for dependencies and assess whether a build's cache is valid. In order to mix build types for this runtime and get a successful SAM build I can deploy, I've had to manually copy `.aws-sam/build/{LogicalFunctionID}/node_modules` over to `.aws-sam/cache/{uuid}/node_modules` after running `sam build --use-container LogicalFunctionID`. After this, `sam build` considers the cache valid and runs an incremental Node build, copying the deps folder into the build directory (equating to an expensive noop since the `node_modules` dir was already there from the container-based build).

**Edit:** there appears to be a change in caching in either SAM 1.91.0 or 1.92.0 - my caching workaround is no longer functioning. In other words, even if you build a layer with `--use-container`, subsequently it isn't considered as being cached by SAM with `sam build` without `--use-container`.

**Edit2:** the caching workaround requires creation of entries in `.aws-sam/build.toml` as per https://github.com/aws/aws-sam-cli/issues/5585#issuecomment-1659384801. Once a build definition is present, builds will update the definition in `build.toml` correctly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.