bottlerocket-os / bottlerocket-os/bottlerocket-core-kit
Flaky builds in AWS CodeBuild
- Dominant language
- Rust
- Stars
- 34
- Forks
- 77
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 23
Description
**Platform I'm building on:**
AWS CodeBuild `ARM_CONTAINER`.
**What I expected to happen:**
Build the kit.
**What actually happened:**
```bash
error: failed to run custom build command for `microcode v0.1.0 (/codebuild/output/src710128580/src/bottlerocket-core-kit/packages/microcode)`
Caused by:
process didn't exit successfully: `/codebuild/output/src710128580/src/bottlerocket-core-kit/target/x86_64/debug/build/microcode-16b683b5bfc6eb34/build-script-build` (exit status: 1)
--- stdout
cargo:rerun-if-env-changed=BUILDSYS_ARCH
cargo:rerun-if-env-changed=BUILDSYS_EXTERNAL_KITS_DIR
cargo:rerun-if-env-changed=BUILDSYS_OUTPUT_GENERATION_ID
cargo:rerun-if-env-changed=BUILDSYS_PACKAGES_DIR
cargo:rerun-if-env-changed=BUILDSYS_ROOT_DIR
cargo:rerun-if-env-changed=BUILDSYS_STATE_DIR
cargo:rerun-if-env-changed=TLPRIVATE_SDK_IMAGE
cargo:rerun-if-changed=Cargo.toml
cargo:rerun-if-changed=/codebuild/output/src710128580/src/bottlerocket-core-kit/build/external-kits/external-kit-metadata.json
cargo:rerun-if-changed=microcode.spec
Error response from daemon: No such image: buildsys-pkg-microcode-x86_64-acca8ab8e27a:latest
Error response from daemon: No such container: buildsys-pkg-microcode-x86_64-acca8ab8e27a-bypass
...
ERROR: failed to solve: frontend grpc server closed unexpectedly
```
**How to reproduce the problem:**
Using a container with all the necessary build prerequisites, build the core kit. This seems to happen very frequently in all of the `ARM_CONTAINER` environments, but far less so on `BUILD_GENERAL1_2XLARGE` (x86_64). I'm not sure if the frequency is because of ARM, or if the 128 vCPUs of the x86_64 2XL somehow mitigates the issue.
1. `git clone https://github.com/bottlerocket-os/bottlerocket-core-kit`
2. `cd bottlerocket-core-kit`
3. `make`
Contributor guide
Research direction
Reproduce with the listed `make` steps in AWS CodeBuild `ARM_CONTAINER`, then inspect the `microcode` package and its custom build command around the reported missing image and container errors. Done means the core kit builds reliably in `ARM_CONTAINER` without these daemon errors; `BUILD_GENERAL1_2XLARGE` provides the comparison described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, rust
- Domain
- build-system, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100