bottlerocket-os / bottlerocket-os/twoliter

slow start-up when not at AWS Seattle offices (I am in Europe)

Open
#421 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
34
Forks
43
Avg merge
11h 13m
Merged PRs (30d)
16

Description

Hi! As mentioned during the community meeting, it takes me somewhere around 15-30 seconds before I get to the build stage. This is at a high level what happens:

### Combined Application and Network Timeline

| Time | Type | Event/Message |
|----------|------|--------------|
| 21:32:35 | App | Command started: `strace -t -f -e trace=network twoliter build variant metal-dev` |
| 21:32:35 | Net | Socket: Initial connection to ECR (44.216.203.94:443) |
| 21:32:35 | Log | INFO twoliter::project::lock - Resolving project references to check against lock file |
| 21:32:35 | Log | INFO twoliter::project::lock::image - Resolving dependency image dependency 'bottlerocket-core-kit-5.0.0@public.ecr.aws/bottlerocket/bottlerocket-core-kit:v5.0.0' |
| 21:32:35 | Net | Socket: Secondary ECR connection (44.216.203.94:443) |
| 21:32:36 | Net | Socket: AWS service connection (75.2.101.78:443) |
| 21:32:36 | Net | Socket: AWS service connection (99.83.145.10:443) |
| 21:32:37 | Net | Socket: Additional ECR connection (44.216.203.94:443) |
| 21:32:37 | Net | DNS: Failed NSCD lookup |
| 21:32:41 | Net | IPv6: Multiple connection attempts failed (ENETUNREACH) |
| 21:32:41 | Log | INFO twoliter::project::lock::image - Resolving dependency image dependency 'bottlerocket-kernel-kit-1.0.0@public.ecr.aws/bottlerocket/bottlerocket-kernel-kit:v1.0.0' |
| 21:32:42 | Net | Socket: Additional AWS service connection (99.83.145.10:443) |
| 21:32:43 | Net | Socket: AWS service connection (75.2.101.78:443) |
| 21:32:44 | Net | Socket: Additional AWS service connection (99.83.145.10:443) |
| 21:32:45 | Net | Socket: Additional ECR connection (44.216.203.94:443) |
| 21:32:46 | Net | Socket: AWS service connection (75.2.101.78:443) |
| 21:32:47 | Net | Socket: AWS service connection (99.83.145.10:443) |
| 21:32:48 | Net | Socket: AWS service connection (3.164.226.167:443) |
| 21:32:48 | Log | INFO twoliter::project::lock::image - Resolving dependency image dependency 'bottlerocket-sdk-0.50.0@public.ecr.aws/bottlerocket/bottlerocket-sdk:v0.50.0' |
| 21:32:48 | Net | IPv6: Multiple failed connection attempts |
| 21:32:49 | Net | Socket: Final ECR connection (44.216.203.94:443) |
| 21:32:50 | Net | Socket: Connection termination begins |
| 21:32:52 | Log | INFO cargo-make 0.37.23 |
| 21:32:52 | Log | WARN Unable to parse Cargo.toml via cargo-metadata, fallbacking |
| 21:32:53 | Log | INFO Build File: /home/mikn/devel/molnett/src/platform/base/bottlerocket/build/tools/Makefile.toml |
| 21:32:53 | Log | INFO Task: build |
| 21:32:53 | Log | INFO Profile: development |
| 21:32:53 | Log | INFO Running Task: setup |
| 21:32:53 | Log | INFO Running Task: setup-build |
| 21:32:53 | Log | INFO Running Task: fetch-sdk |

I had this strace output analysed by an AI and produced this log - but this was of a broken run as you can see by the fact that Cargo.toml was not parse:able.

Perhaps we could at least do all of the images in parallel?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing `twoliter build variant metal-dev` and tracing the project lock image-resolution stage described in the logs. Inspect the referenced `src/platform/base/bottlerocket/build/tools/Makefile.toml` and the Cargo.toml parsing path. Done should mean dependency images resolve without the reported startup delay and the build reaches its normal stages successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
build-system, cloud, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.