bottlerocket-os / bottlerocket-os/bottlerocket-core-kit

Flaky builds in AWS CodeBuild

Open
#138 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
34
Forks
77
Avg merge
2d 23h
Merged PRs (30d)
23

Description

**Platform I'm building on:**

AWS CodeBuild `ARM_CONTAINER`.

**What I expected to happen:**

Build the kit.

**What actually happened:**

```bash
error: failed to run custom build command for `microcode v0.1.0 (/codebuild/output/src710128580/src/bottlerocket-core-kit/packages/microcode)`
Caused by:
process didn't exit successfully: `/codebuild/output/src710128580/src/bottlerocket-core-kit/target/x86_64/debug/build/microcode-16b683b5bfc6eb34/build-script-build` (exit status: 1)
--- stdout
cargo:rerun-if-env-changed=BUILDSYS_ARCH
cargo:rerun-if-env-changed=BUILDSYS_EXTERNAL_KITS_DIR
cargo:rerun-if-env-changed=BUILDSYS_OUTPUT_GENERATION_ID
cargo:rerun-if-env-changed=BUILDSYS_PACKAGES_DIR
cargo:rerun-if-env-changed=BUILDSYS_ROOT_DIR
cargo:rerun-if-env-changed=BUILDSYS_STATE_DIR
cargo:rerun-if-env-changed=TLPRIVATE_SDK_IMAGE
cargo:rerun-if-changed=Cargo.toml
cargo:rerun-if-changed=/codebuild/output/src710128580/src/bottlerocket-core-kit/build/external-kits/external-kit-metadata.json
cargo:rerun-if-changed=microcode.spec
Error response from daemon: No such image: buildsys-pkg-microcode-x86_64-acca8ab8e27a:latest
Error response from daemon: No such container: buildsys-pkg-microcode-x86_64-acca8ab8e27a-bypass

...

ERROR: failed to solve: frontend grpc server closed unexpectedly
```

**How to reproduce the problem:**

Using a container with all the necessary build prerequisites, build the core kit. This seems to happen very frequently in all of the `ARM_CONTAINER` environments, but far less so on `BUILD_GENERAL1_2XLARGE` (x86_64). I'm not sure if the frequency is because of ARM, or if the 128 vCPUs of the x86_64 2XL somehow mitigates the issue.

1. `git clone https://github.com/bottlerocket-os/bottlerocket-core-kit`
2. `cd bottlerocket-core-kit`
3. `make`

Contributor guide

Open the contributing guide

Research direction

Reproduce with the listed `make` steps in AWS CodeBuild `ARM_CONTAINER`, then inspect the `microcode` package and its custom build command around the reported missing image and container errors. Done means the core kit builds reliably in `ARM_CONTAINER` without these daemon errors; `BUILD_GENERAL1_2XLARGE` provides the comparison described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, rust
Domain
build-system, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.