aws-amplify / aws-amplify/amplify-hosting

Layer packaging failed but Build still "succeeded" causing outage

Open
#2,988 1 comment 0 reactions 0 assignees View on GitHub
backend-builds bug
Dominant language
Dockerfile
Stars
481
Forks
123
PR merge metrics
No merged PRs in 30d

Description

### Before opening, please confirm:

- [X] I have checked to see if my question is addressed in the [FAQ](https://github.com/aws-amplify/amplify-hosting/blob/master/FAQ.md).
- [X] I have [searched for duplicate or closed issues](https://github.com/aws-amplify/amplify-hosting/issues?q=is%3Aissue+).
- [X] I have read the guide for [submitting bug reports](https://github.com/aws-amplify/amplify-hosting/blob/master/CONTRIBUTING.md).
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

### App Id

d2d6ty1wlg8jch

### Region

us-east-1

### Amplify Hosting feature

Backend builds

### Describe the bug

I triggered a build, and an outage occured with one of our Lambdas not able to find packages that are installed via a Lambda Layer.

Looking at the build logs I see this error that seems to be the cause. AWS Amplify console however continued with the build, marked it as successful, and deployed it, causing an outage in our system. We rolled back to previous build and the outage was resolved.

AWS Amplify console 2022-09-15 13-45-30

### Expected behavior

The build should have failed, preventing an incorrect deployment.

### Reproduction steps

Seems like there was a temp network issue which caused this.

### Build Settings

```yaml
I don't think this is relevant.
```

### Additional information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked issue details and the reported build logs for the Lambda Layer packaging error. Reproduce the failure caused by the temporary network issue if possible, then trace how backend builds interpret that error. Done means a failed layer-packaging step prevents deployment and the build is reported as unsuccessful.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
ci-cd, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.