INFO: Collecting Logs on Bottlerocket AMI

# Bottlerocket log collection

### No host logs on Bottlerocket

UPDATE: please see new workaround from community: https://github.com/aws-samples/amazon-cloudwatch-container-insights/issues/136#issuecomment-2143989603

The Bottlerocket AMIs are meant to be a very stripped down container OS. Consequently, there are fewer log types to collect.

I've reached out the bottlerocket team and they said that all logs should go to journald on Bottlerocket.

On Bottlerocket, the `/aws/containerinsights/Cluster_Name/host` will not be populated because `/var/log/dmesg`, `/var/log/secure`, and `/var/log/messages` files do not exist on bottlerocket.

dmesg logs can be obtained with `journalctl -k` or `journalctl --dmesg`:

```
bash-5.1# journalctl -k
May 19 19:21:55 localhost kernel: Linux version 5.15.108 (builder@buildkitsandbox) (x86_64-bottlerocket-linux-gnu-gcc (Buildroot 2022.11.1) 11.3.0, GNU ld (GNU Binutils) 2.38) #1 SMP Tue May 9 23:54:36 UTC 2023
May 19 19:21:55 localhost kernel: Command line: BOOT_IMAGE=(hd0,gpt3)/vmlinuz console=tty0 console=ttyS0,115200n8 net.ifnames=0 netdog.default-interface=eth0:dhcp4,dhcp6? quiet bootconfig root=/dev/dm-0 rootwait ro raid=noautodetect random.trust_cpu=on selinux=1 enforcing=1 "dm-mod.create=root,,,ro,0 1884160 verity 1 PARTUUID=9b48037e-3bab-4072-8daf-082d8fae9f5e/PARTNROFF=1 PARTUUID=9b48037e-3bab-4072-8daf-082d8fae9f5e/PARTNROFF=2 4096 4096 235520 1 sha256 f12f74a243a23635effccde072228bc55a9f06c4c6e587001619bff01f6f8a16 80248401eae3104a1d94e19d04da0bc2b90289d599845063bdf6b330c422e713 2 restart_on_corruption ignore_zero_blocks" -- systemd.log_target=journal-or-kmsg systemd.log_color=0 systemd.show_status=true
May 19 19:21:55 localhost kernel: KASLR enabled
May 19 19:21:55 localhost kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
May 19 19:21:55 localhost kernel: x86/fpu: Supporting XSAVE
```

However, Fluent Bit appears to only be able to collect systemd unit journald logs. The docs say that it can filter by systemd unit file only: https://docs.fluentbit.io/manual/pipeline/inputs/systemd

And when I tried an input with no filters, no logs were collected:

```
[INPUT]
Name systemd
Tag systemd.all.*
DB /var/fluent-bit/state/systemd.db
Read_From_Tail Off
```

Here's the contents of `/var/log` on my node:

```
[ec2-user@admin]$ sudo sheltie
bash-5.1# ls /var/log
aws-routed-eni containers dmesg journal kdump pods private support
```

The logs in `aws-routed-eni` might be interesting to some users and could be collected with a Tail input: https://docs.fluentbit.io/manual/pipeline/inputs/tail

```
bash-5.1# cd aws-routed-eni/
bash-5.1# ls
egress-v4-plugin.log ipamd.log plugin.log
```

Notes:

* There is a dmesg directory on my node in `/var/log`, but its empty. I've reached out the bottlerocket team and they said that all logs should go to journald on Bottlerocket. As noted above, you can use `journalctl` to obtain them.
* Make sure you use the admin container if you choose to poke around the filesystem. I found that it seemed some directories were not visible without admin.

### Pod Logs will still be collected

Pod logs can still be collected without any change in experience.

### Dataplane logs will still be collected

Kubelet and Containerd logs can still be collected without any change in experience.

```
[INPUT]
Name systemd
Tag dataplane.systemd.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Systemd_Filter _SYSTEMD_UNIT=containerd.service
DB /var/fluent-bit/state/systemd.db
Path /var/log/journal
Read_From_Tail ${READ_FROM_TAIL}
```

Start with the Bottlerocket log paths and the Fluent Bit systemd and tail input examples in the issue. Compare the documented host, pod, and dataplane collection behavior with the linked community workaround. Done means the Bottlerocket limitations and supported collection approach are clearly documented without implying that unavailable host log files will be populated.

Written by the indexing model from the issue text.