Amplify CI/CD Builds are Broken and Unreliable

### Before opening, please confirm:

- [X] I have checked to see if my question is addressed in the [FAQ](https://github.com/aws-amplify/amplify-hosting/blob/master/FAQ.md).
- [X] I have [searched for duplicate or closed issues](https://github.com/aws-amplify/amplify-hosting/issues?q=is%3Aissue+).
- [X] I have read the guide for [submitting bug reports](https://github.com/aws-amplify/amplify-hosting/blob/master/CONTRIBUTING.md).
- [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

### App Id

d3rtqmm3xlsvys

### Region

us-west-2

### Amplify Hosting feature

Backend builds, Build settings, Custom builds

### Describe the bug

Builds are completely erratic and unpredictable when you have prior broken builds.

The same `amplifyPush` command (and its hooks) run multiple times in the CI/CD pipeline, totally inexplicably.

I am attaching a github repo with its commit history + each of the CI/CD build logs that go with the commits.

You will see as the CI/CD pipeline grows increasingly erratic/broken.

This is a critical, serious bug. I've now reproduced it on 3 different Amplify applications - and frankly, I worry about doing ANYTHING that's mission-critical or customer-facing with this kind of instability in the build pipeline.

Please see the "Reproduction steps" below for a full history of the code, the commits, and the build logs for each commit.

### Expected behavior

As a customer, I expect that:

- CI/CD pipelines run idempotently
- CI/CD pipelines run atomically
- CI/CD pipelines run reproducibly
- AmplifyPush gets triggered ONLY once, when executed/called only once in the build spec (`amplify.yml`)
- That the commands and steps executed in the CI/CD pipeline **strictly** conform to EXACTLY the steps in `amplify.yml` without any unexpected or duplicate command/step executions

### Reproduction steps

My reproduction repo: https://github.com/armenr/amplify-bug-repro-2

## Full reproduction steps

1. degit antfu/vitesse bug-repro-amplify (and DO NOT PUSH TO GIT YET)
2. create some repo in GitHub (but don't commit or push to your repo yet)
3. `cd vitesse bug-repro-amplify && npx taze major && pnpm i`
4. DO NOT COMMIT ANYTHING YET
5. copy the following from my repo into your new repo
- `amplify.yml`
- `scripts/*` (everything in the scripts folder)
6. Run `./scripts/init-headless.sh `git commit -am "initial commit" && git push`
13. Watch build fail
14. add the following to your code
- `"lint:fix": "eslint . --fix",` --> add to `package.json`
15. commit and push this change
16. watch the build fail
17. copy the following from my repo into your new repo
- `.eslintignore`
18. commit and push this change
19. Watch build fail
14. add the following to your code
- `"build:ssg": "vite-ssg build",` --> add to `package.json`
- `"build:std": "vite build",` --> add to `package.json`
15. commit and push this change
16. Watch the build succeed, but OBSERVE THE NUMBER OF TIMES YOU SEE THE AMPLIFY HOOKS RUN! (See logs from the last commit step below)

## BUILD LOGS

### Really concerning things:
1. The string "STEP ONE: [Pre-PULL] script running!" occurs only ONE time but...
2. The string "STEP ONE: [Pre-Push] script running!" occurs ONE time but...
3. The string "STEP TWO: [Post-Pull] script running!" occurs TWICE
4. The string "STEP TWO: [Post-Push] script running!" occurs TWICE

```
2022-08-30T04:32:55.050Z [INFO]: # Cloning repository: git@github.com:armenr/amplify-bug-repro-2.git
2022-08-30T04:32:55.646Z [INFO]: Cloning into 'amplify-bug-repro-2'...
2022-08-30T04:32:55.647Z [INFO]: # Switching to commit: b7e147e1889cb038fc54382a400fd2d0af486b2a
2022-08-30T04:32:55.654Z [INFO]: Note: switching to 'b7e147e1889cb038fc54382a400fd2d0af486b2a'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at b7e147e Fix build scripts
2022-08-30T04:32:55.683Z [INFO]: Successfully cleaned up Git credentials
2022-08-30T04:32:55.683Z [INFO]: # Checking for Git submodules at: /codebuild/output/src500293666/src/amplify-bug-repro-2/.gitmodules
2022-08-30T04:32:55.710Z [INFO]: # Retrieving environment cache...
2022-08-30T04:32:55.761Z [INFO]: # Retrieved environment cache
2022-08-30T04:32:55.762Z [INFO]: ---- Setting Up SSM Secrets ----
2022-08-30T04:32:55.762Z [INFO]: SSM params {"Path":"/amplify/d3rtqmm3xlsvys/main/","WithDecryption":true}
2022-08-30T04:32:56.847Z [WARNING]: Cannot find module '/root/.nvm/versions/node/none/lib/node_modules/@aws-amplify/cli/package.json'
Require stack:
- /snapshot/AWSMobilePioneerExecute/src/plugins/patch/npm.js
- /snapshot/AWSMobilePioneerExecute/src/plugins/patch/patch.js
- /snapshot/AWSMobilePioneerExecute/pioneer.js
2022-08-30T04:32:56.847Z [INFO]: # Patching NPM package '@aws-amplify/cli' to 9.2.1...
2022-08-30T04:32:59.501Z [INFO]: # Done patching NPM package '@aws-amplify/cli' to version 9.2.1
2022-08-30T04:33:07.851Z [INFO]: ## Starting Backend Build
# Starting phase: preBuild
2022-08-30T04:33:07.853Z [INFO]: # Executing command: aws configure set cli_follow_urlparam false
2022-08-30T04:33:10.000Z [INFO]: # Executing command: export AWS_BRANCH_LONG_NAME=main
2022-08-30T04:33:10.005Z [INFO]: # Executing command: export AWS_BRANCH=main
2022-08-30T04:33:10.007Z [INFO]: # Executing command: npm install
2022-08-30T04:33:34.348Z [WARNING]: npm
2022-08-30T04:33:34.350Z [WARNING]: WARN deprecated libnpmconfig@1.2.1: This module is not used anymore. npm config is parsed by npm itself and by @npmcli/config
2022-08-30T04:33:49.586Z [INFO]: added 1124 packages, and audited 1125 packages in 39s
2022-08-30T04:33:49.589Z [INFO]: 214 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
2022-08-30T04:33:49.607Z [INFO]: # Completed phase: preBuild
# Starting phase: build
2022-08-30T04:33:49.608Z [INFO]: # Executing command: echo "[!] Executing Amplify CLI helper script"
[!] Executing Amplify CLI helper script
2022-08-30T04:33:50.734Z [INFO]:
2022-08-30T04:33:50.734Z [INFO]: ----- 🪝 pre-pull execution start -----
2022-08-30T04:33:50.748Z [INFO]: STEP ONE: [Pre-PULL] script running!
Stashing package.json
Removing type: module and writing modified package.json
2022-08-30T04:33:50.750Z [INFO]: ----- 🪝 pre-pull execution end -----
2022-08-30T04:33:50.750Z [INFO]:
2022-08-30T04:33:51.771Z [INFO]: Amplify AppID found: d3rtqmm3xlsvys. Amplify App name is: amplify-bug-repro-2
2022-08-30T04:33:51.850Z [INFO]: Backend environment main found in Amplify Console app: amplify-bug-repro-2
2022-08-30T04:33:52.695Z [WARNING]: - Fetching updates to backend environment: main from the cloud.
2022-08-30T04:33:53.510Z [WARNING]: ✔ Successfully pulled backend environment main from the cloud.
2022-08-30T04:33:53.519Z [INFO]: ✅
2022-08-30T04:33:53.523Z [INFO]:
2022-08-30T04:33:53.523Z [INFO]: ----- 🪝 post-pull execution start -----
2022-08-30T04:33:53.536Z [INFO]: STEP TWO: [Post-Pull] script running!
2022-08-30T04:33:53.541Z [INFO]: Porting js exports to ts...
2022-08-30T04:33:53.547Z [INFO]: Linting & fixing ts exports file...
2022-08-30T04:33:53.782Z [INFO]: > lint:fix
> eslint . --fix
2022-08-30T04:33:56.368Z [INFO]: Restoring package.json
2022-08-30T04:33:56.371Z [INFO]: ----- 🪝 post-pull execution end -----
2022-08-30T04:33:57.660Z [INFO]:
2022-08-30T04:33:57.660Z [INFO]: ----- 🪝 pre-push execution start -----
2022-08-30T04:33:57.673Z [INFO]: STEP ONE: [Pre-Push] script running!
Checking if amplify config files exist
amplify config files do exist!
2022-08-30T04:33:57.676Z [INFO]: checking if aws-exports.js exists
2022-08-30T04:33:57.677Z [INFO]: aws-exports.js does exist
Moving aws-exports.js to aws-exports.cjs to avoid nasty 'type: module' bug
2022-08-30T04:33:57.680Z [INFO]: ----- 🪝 pre-push execution end -----
2022-08-30T04:33:57.733Z [INFO]: Note: It is recommended to run this command from the root of your app directory
2022-08-30T04:33:58.282Z [WARNING]: - Initializing your environment: main
2022-08-30T04:33:59.087Z [WARNING]: ✔ Initialized provider successfully.
2022-08-30T04:33:59.384Z [WARNING]: - Updating resources in the cloud. This may take a few minutes...
2022-08-30T04:34:00.358Z [INFO]: ✅ Initialized your environment successfully.
2022-08-30T04:34:00.360Z [INFO]: Your project has been successfully initialized and connected to the cloud!
Some next steps:
"amplify status" will show you what you've added already and if it's locally configured or deployed
"amplify add " will allow you to add features like user login or a backend API
"amplify push" will build all your local backend resources and provision it in the cloud
"amplify console" to open the Amplify Console and view your project status
"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud
Pro tip:
Try "amplify add api" to create a backend API and then "amplify push" to deploy everything
⚠️ The maximum number of apps that you can create with Amplify in this region has likely been reached:
For more information on Amplify Service Quotas, see:
https://docs.aws.amazon.com/general/latest/gr/amplify.html#service-quotas-amplify
2022-08-30T04:34:00.361Z [INFO]: ----- 🪝 post-push execution start -----
2022-08-30T04:34:00.379Z [INFO]: STEP TWO: [Post-Push] script running!
2022-08-30T04:34:00.380Z [INFO]: Porting js exports to ts...
2022-08-30T04:34:00.386Z [INFO]: Linting & fixing ts exports file...
2022-08-30T04:34:03.186Z [INFO]: ----- 🪝 post-push execution end -----
2022-08-30T04:34:03.187Z [INFO]: ----- 🪝 post-push execution start -----
2022-08-30T04:34:03.199Z [INFO]: STEP TWO: [Post-Push] script running!
2022-08-30T04:34:03.204Z [INFO]: Porting js exports to ts...
2022-08-30T04:34:03.210Z [INFO]: Linting & fixing ts exports file...
2022-08-30T04:34:05.989Z [INFO]: ----- 🪝 post-push execution end -----
2022-08-30T04:34:07.591Z [INFO]: # Completed phase: build
## Completed Backend Build
2022-08-30T04:34:07.594Z [INFO]: ## Starting Frontend Build
# Starting phase: preBuild
2022-08-30T04:34:07.594Z [INFO]: # Executing command: git status
2022-08-30T04:34:07.598Z [INFO]: HEAD detached at b7e147e
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git restore ..." to discard changes in working directory)
modified: amplify/team-provider-info.json
Untracked files:
(use "git add ..." to include in what will be committed)
amplify.sh
package-lock.json
src/aws-exports.ts
no changes added to commit (use "git add" and/or "git commit -a")
2022-08-30T04:34:07.598Z [INFO]: # Executing command: echo "Git checkout from inside frontend preBuild phase"
Git checkout from inside frontend preBuild phase
2022-08-30T04:34:07.598Z [INFO]: # Executing command: git checkout package.json
2022-08-30T04:34:07.599Z [WARNING]: Updated 0 paths from the index
2022-08-30T04:34:07.600Z [INFO]: # Executing command: echo "Testing for aws-exports!!"
2022-08-30T04:34:07.600Z [INFO]: Testing for aws-exports!!
# Executing command: test -f src/aws-exports.ts && cat src/aws-exports.ts
2022-08-30T04:34:07.601Z [INFO]: // !! WARNING: This file is autogenerated. DO NOT MAKE CHANGES BY HAND!
const awsmobile = {
aws_project_region: 'us-west-2',
}
export default awsmobile
2022-08-30T04:34:07.601Z [INFO]: # Completed phase: preBuild
# Starting phase: build
# Executing command: git status
2022-08-30T04:34:07.603Z [INFO]: HEAD detached at b7e147e
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git restore ..." to discard changes in working directory)
modified: amplify/team-provider-info.json
Untracked files:
(use "git add ..." to include in what will be committed)
amplify.sh
package-lock.json
src/aws-exports.ts
no changes added to commit (use "git add" and/or "git commit -a")
2022-08-30T04:34:07.603Z [INFO]: # Executing command: echo "Git checkout from inside frontend build phase"
Git checkout from inside frontend build phase
# Executing command: git checkout package.json
2022-08-30T04:34:07.604Z [WARNING]: Updated 0 paths from the index
2022-08-30T04:34:07.605Z [INFO]: # Executing command: npm run build:ssg
2022-08-30T04:34:07.840Z [INFO]: > build:ssg
> vite-ssg build
2022-08-30T04:34:09.519Z [INFO]: [vite-ssg] Build for client...
2022-08-30T04:34:09.666Z [INFO]: vite v3.0.9 building for production...
2022-08-30T04:34:09.704Z [INFO]: transforming...
2022-08-30T04:34:11.573Z [INFO]: ✓ 87 modules transformed.
2022-08-30T04:34:11.834Z [INFO]: rendering chunks...
2022-08-30T04:34:12.054Z [INFO]: dist/index.html 0.93 KiB
2022-08-30T04:34:12.054Z [INFO]: dist/manifest.webmanifest 0.38 KiB
2022-08-30T04:34:12.055Z [INFO]: dist/ssr-manifest.json 3.58 KiB
2022-08-30T04:34:12.056Z [INFO]: dist/assets/404.8e6ee3ca.js 0.49 KiB / gzip: 0.35 KiB
2022-08-30T04:34:12.056Z [INFO]: dist/assets/_...all_.1faa463a.js 0.23 KiB / gzip: 0.20 KiB
dist/assets/README.1f91b565.js 3.45 KiB / gzip: 0.81 KiB
2022-08-30T04:34:12.056Z [INFO]: dist/assets/about.30b304bd.js 3.81 KiB / gzip: 1.05 KiB
dist/assets/home.3f0c633b.js 0.38 KiB / gzip: 0.28 KiB
2022-08-30T04:34:12.057Z [INFO]: dist/assets/index.f5da2381.css 18.28 KiB / gzip: 4.78 KiB
2022-08-30T04:34:12.057Z [INFO]: dist/assets/virtual_pwa-register.9b01f6a2.js 5.52 KiB / gzip: 2.31 KiB
2022-08-30T04:34:12.061Z [INFO]: dist/assets/_name_.95a91a20.js 0.97 KiB / gzip: 0.57 KiB
2022-08-30T04:34:12.065Z [INFO]: dist/assets/app.ea254bd1.js 148.34 KiB / gzip: 53.80 KiB
2022-08-30T04:34:13.961Z [INFO]: PWA v0.12.3
mode generateSW
precache 15 entries (182.40 KiB)
files generated
dist/sw.js
dist/workbox-3ea082d2.js
2022-08-30T04:34:13.962Z [INFO]: [vite-ssg] Build for server...
2022-08-30T04:34:14.150Z [INFO]: vite v3.0.9 building SSR bundle for production...
2022-08-30T04:34:14.166Z [INFO]: transforming...
2022-08-30T04:34:14.662Z [INFO]: ✓ 48 modules transformed.
2022-08-30T04:34:14.754Z [INFO]: rendering chunks...
2022-08-30T04:34:14.835Z [INFO]: .vite-ssg-temp/manifest.webmanifest 0.38 KiB
2022-08-30T04:34:14.835Z [INFO]: .vite-ssg-temp/main.mjs 87.39 KiB
.vite-ssg-temp/assets/virtual_pwa-register.9b01f6a2.js 8.13 KiB
.vite-ssg-temp/assets/README.ceaa998f.js 4.99 KiB
.vite-ssg-temp/assets/_...all_.6eb20659.js 1.07 KiB
.vite-ssg-temp/assets/about.085950da.js 5.42 KiB
.vite-ssg-temp/assets/_name_.aced8ca1.js 2.50 KiB
.vite-ssg-temp/assets/style.fed351e7.css 22.62 KiB
2022-08-30T04:34:14.894Z [INFO]: [vite-ssg] Rendering Pages... (3)
2022-08-30T04:34:14.930Z [INFO]: [vite-ssg] Critical CSS generation enabled via `critters`
2022-08-30T04:34:15.380Z [INFO]: dist/index.html 10.27 KiB
2022-08-30T04:34:15.381Z [INFO]: dist/readme.html 13.09 KiB
dist/about.html 13.20 KiB
2022-08-30T04:34:15.382Z [INFO]: [vite-ssg] Regenerate PWA...
2022-08-30T04:34:16.438Z [INFO]: PWA v0.12.3
mode generateSW
precache 17 entries (218.03 KiB)
files generated
dist/sw.js
dist/workbox-3ea082d2.js
[vite-ssg] Build finished.
2022-08-30T04:34:16.502Z [INFO]: # Completed phase: build
## Completed Frontend Build
2022-08-30T04:34:16.522Z [INFO]: ## Build completed successfully
2022-08-30T04:34:16.526Z [INFO]: !!Invalid custom header config file. The `customHeaders` key should be an array
2022-08-30T04:34:16.597Z [INFO]: # Starting build artifact upload process...
2022-08-30T04:34:16.702Z [INFO]: # Build artifact is: 0MB
2022-08-30T04:34:16.702Z [INFO]: # Uploading build artifact '__artifacts.zip'...
2022-08-30T04:34:16.703Z [INFO]: # Build artifact is: 0MB
2022-08-30T04:34:16.703Z [INFO]: # Uploading build artifact '__artifactsHash.zip'...
2022-08-30T04:34:16.793Z [INFO]: # Build artifact upload completed
2022-08-30T04:34:16.793Z [INFO]: # Starting environment caching...
2022-08-30T04:34:16.794Z [INFO]: # Uploading environment cache artifact...
2022-08-30T04:34:16.873Z [INFO]: # Environment caching completed
Terminating logging...

```

### Build Settings

```yaml
## Custom Amplify settings
## REF: https://docs.aws.amazon.com/amplify/latest/userguide/build-settings.html
## TODO: MIGRATE TO MONOREPO APPROOT SETUP
## https://docs.aws.amazon.com/amplify/latest/userguide/monorepo-configuration.html
version: 1
# env:
# variables:
# AMPLIFY_DIFF_BACKEND: false
# AMPLIFY_DIFF_DEPLOY: false
# AWS_CACHE_BUCKET_READ: false
# USER_BRANCH:

# BEGIN backend
backend:
phases:
preBuild:
commands:
# - echo "Setting PNPM store for package caching!"
# - pnpm config set store-dir /root/.pnpm-store
# fix for parameter store when storing URLs: https://github.com/aws/aws-cli/issues/2507
- aws configure set cli_follow_urlparam false
# prevent common git branch names (feat/something) from creating invalid stack/resource name
# and set max length to 255 as per: https://docs.aws.amazon.com/amplify/latest/APIReference/API_BackendEnvironment.html
- export AWS_BRANCH_LONG_NAME=$(echo ${AWS_BRANCH} | tr "/" "-"| cut -c1-255)
# prevent long git branch names from creating an invalid stack/resource name
# set to max 20 length as per: https://github.com/aws-samples/aws-serverless-airline-booking/blob/main/amplify.yml#L9
- export AWS_BRANCH=$(echo ${AWS_BRANCH_LONG_NAME} | tr "/" "-"| cut -c1-20)
- npm install
build:
commands:
- echo "[!] Executing Amplify CLI helper script"
- amplifyPush --simple

## BEGIN FrontEnd
frontend:
phases:
preBuild:
commands:
- git status
- echo "Git checkout from inside frontend preBuild phase"
- git checkout package.json
- echo "Testing for aws-exports!!"
- test -f src/aws-exports.ts && cat src/aws-exports.ts
# - pnpm config set store-dir /root/.pnpm-store
# - pnpm install
build:
commands:
- git status
- echo "Git checkout from inside frontend build phase"
- git checkout package.json
- npm run build:ssg
artifacts:
baseDirectory: dist
files:
- "**/*"
# cache:
# paths:
# - /root/.pnpm-store/**/*
# - node_modules/**/*

```
```

### Additional information

_No response_

Start with the linked reproduction repository, especially amplify.yml and scripts/*, then compare its commit history with the attached CI/CD build logs. Reproduce the duplicate hook executions and trace which build steps diverge from amplify.yml; done means each declared command runs once and the pipeline is reproducible.

Written by the indexing model from the issue text.