aws / aws/ec2-image-builder-roadmap

[Bug] [Container Image Builder] When AWSTOE fails, it says logs can be found in /var/lib/amazon/toe/ but folder does not exist.

Open
#91 0 comments 0 reactions 0 assignees View on GitHub
Proposed
Dominant language
No language data
Stars
35
Forks
7
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Build a container recipe.
Not sure this is relevant, but I'm using "Amazon Linux AMI 2023.0.20230627 x86_64 ECS HVM EBS" (ami-0a6c1a11150f7af05 in eu-central-1) as the bage image for my build machine.

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
When AWSTOE fails for whatever reason TOE_LOGS in /tmp/imagebuilder_service/TOE_LOGS look like this
```
{
"executionId": "3b1ce2ae-1687-11ee-a8af-0242ac110002",
"status": "failed",
"failedStepCount": 1,
"executedStepCount": 1,
"ignoredFailedStepCount": 0,
"failureMessage": "Document arn:aws:imagebuilder:eu-central-1:454249771573:component/mariadb-linux/1.0.0/1 failed!",
"logUrl": "/var/lib/amazon/toe/TOE_2023-06-29_14-14-16_UTC-0_3b1ce2ae-1687-11ee-a8af-0242ac110002"
}
```

Problem is: folder /var/lib/amazon/toe/ does not even exist.

**Are you currently working around this issue?**
As a consequence, I needed to manually replay TOE after connecting to the builder VM.

```
## 1. Install AWSTOE

sudo curl https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/386/awstoe -o /usr/bin/awstoe

sudo chmod +x /usr/bin/awstoe

## 2. Download component data

sudo touch document.yml
sudo chmod 777 document.yml
sudo aws imagebuilder get-component --component-build-version-arn arn:aws:imagebuilder:eu-central-1:aws:component/mariadb-linux/1.0.0/1 --query "component.data" --output text > document.yml

## 3. Run document

sudo su
awstoe validate --documents document.yml

{
"validationStatus": "success",
"message": "Document(s) [document.yml] is/are valid."
}

awstoe run --documents document.yml
{
"executionId": "af3dc7dc-1692-11ee-b11f-0a750e1b65ac",
"status": "success",
"failedStepCount": 0,
"executedStepCount": 3,
"ignoredFailedStepCount": 0,
"failureMessage": "",
"logUrl": "/tmp/imagebuilder_service/TOE_2023-06-29_15-36-15_UTC-0_af3dc7dc-1692-11ee-b11f-0a750e1b65ac"
}
```

Then as you can see logs are located in /tmp/imagebuilder_service/ (and there they DO exist :)

**Additional context**
Not sure this is relevant, but I'm using
- Amazon Linux 2023 AMI (cf. above)
- I have the issue with all base docker images I tried (one is based on Ubuntu, the other one on AlmaLinux minimal)

Contributor guide

Open the contributing guide

Research direction

Start with the failed AWSTOE result in /tmp/imagebuilder_service/TOE_LOGS and compare its logUrl with the actual logs produced by awstoe run. Reproduce the failure on the described Amazon Linux 2023 builder and verify that the reported log path exists and points to the available logs instead of /var/lib/amazon/toe/.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.