aws / aws/amazon-ecs-cni-plugins
Missing and truncated `ecs-bridge` logs
- Dominant language
- Go
- Stars
- 101
- Forks
- 28
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
We're running observing `ecs-bridge` logs for different ECS Tasks being truncated.
This always presents as the earlier logs for a given container being missing and/or having some information missing from the start of the log message.
### Detail
The following example logs are taken from one EC2 instance in an ECS Cluster. The instance had just booted and joined the cluster. This was part of an instance refresh, so this new instance received several tasks from other instances that were being drained. This is the first few seconds of the log file, but enough to demonstrate the issue. No log rotation has happened, as the instance has only been running a very short time.
I've replaced the `containerID` and `hostVentName` fields with simple, but consistent values to make the messages a little easier to read.
While https://github.com/cihub/seelog hasn't updates for a while, I note there are two relating to truncating logs.
### Expected behaviour
1. No logs are truncated.
1. All expected logs are present.
This is demonstrated in the log below for `containerID=1`, `containerID=4` and `containerID=5`. A normal sequence of messages is present - "Creating the bridge", "Creating veth pair for namespace", "Attaching veth pair to bridge", "Running IPAM plugin ADD", "Configuring container's interface" and finally "Configuring bridge".
### Actual behaviour
1. For `containerID=2`
1. , the first log message is missing the timestamp, severity, `msg` field and the `netns=` part of the `netns` field.
1. The log message begins with the value of the `netns` field, followed by the remainder of the line.
1. This appears to be the "Running IPAM plugin ADD" log message.
1. The subsequent "Configuring container's interface" and final "Configuring bridge" messages appear to be logged normally
1. The "Creating the bridge", "Creating veth pair for namespace" and "Attaching veth pair to bridge" messages are completely missing
1. For `containerID=3`
1. The log messages that are written appear complete
1. The "Creating the bridge" and "Creating veth pair for namespace" are missing
### Sample log
```
2025-06-09T07:51:54Z [INFO] msg="Creating the bridge" netns=/host/proc/2334/ns/net ifname=ecs-eth0 containerID=1 bridgeName=ecs-bridge ipamType=ecs-ipam
2025-06-09T07:51:54Z [INFO] Setting ecs-bridge hardware addr (MAC) 7a:c0:ff:b6:64:4b
2025-06-09T07:51:54Z [INFO] msg="Creating veth pair for namespace" netns=/host/proc/2334/ns/net ifname=ecs-eth0 containerID=1 bridgeName=ecs-bridge ipamType=ecs-ipam
2025-06-09T07:51:54Z [INFO] msg="Attaching veth pair to bridge" netns=/host/proc/2334/ns/net ifname=ecs-eth0 containerID=1 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth1
2025-06-09T07:51:54Z [INFO] msg="Running IPAM plugin ADD" netns=/host/proc/2334/ns/net ifname=ecs-eth0 containerID=1 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth1
2025-06-09T07:51:54Z [INFO] msg="Configuring container's interface" netns=/host/proc/2334/ns/net ifname=ecs-eth0 containerID=1 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth1
2025-06-09T07:51:54Z [INFO] msg="Configuring bridge" netns=/host/proc/2334/ns/net ifname=ecs-eth0 containerID=1 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth1
/proc/2390/ns/net ifname=ecs-eth0 containerID=2 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth2
2025-06-09T07:51:54Z [INFO] msg="Attaching veth pair to bridge" netns=/host/proc/2352/ns/net ifname=ecs-eth0 containerID=3 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth3
2025-06-09T07:51:54Z [INFO] msg="Running IPAM plugin ADD" netns=/host/proc/2352/ns/net ifname=ecs-eth0 containerID=3 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth3
2025-06-09T07:51:54Z [INFO] msg="Configuring container's interface" netns=/host/proc/2390/ns/net ifname=ecs-eth0 containerID=2 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth2
2025-06-09T07:51:54Z [INFO] msg="Configuring bridge" netns=/host/proc/2390/ns/net ifname=ecs-eth0 containerID=2 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth2
2025-06-09T07:51:54Z [INFO] msg="Configuring container's interface" netns=/host/proc/2352/ns/net ifname=ecs-eth0 containerID=3 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth3
2025-06-09T07:51:54Z [INFO] msg="Configuring bridge" netns=/host/proc/2352/ns/net ifname=ecs-eth0 containerID=3 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth3
2025-06-09T07:51:55Z [INFO] msg="Creating the bridge" netns=/host/proc/2865/ns/net ifname=ecs-eth0 containerID=4 bridgeName=ecs-bridge ipamType=ecs-ipam
2025-06-09T07:51:55Z [INFO] msg="Creating veth pair for namespace" netns=/host/proc/2865/ns/net ifname=ecs-eth0 containerID=4 bridgeName=ecs-bridge ipamType=ecs-ipam
2025-06-09T07:51:55Z [INFO] msg="Attaching veth pair to bridge" netns=/host/proc/2865/ns/net ifname=ecs-eth0 containerID=4 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth4
2025-06-09T07:51:55Z [INFO] msg="Running IPAM plugin ADD" netns=/host/proc/2865/ns/net ifname=ecs-eth0 containerID=4 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth4
2025-06-09T07:51:56Z [INFO] msg="Configuring container's interface" netns=/host/proc/2865/ns/net ifname=ecs-eth0 containerID=4 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth4
2025-06-09T07:51:56Z [INFO] msg="Configuring bridge" netns=/host/proc/2865/ns/net ifname=ecs-eth0 containerID=4 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth4
2025-06-09T07:51:57Z [INFO] msg="Creating the bridge" netns=/host/proc/3399/ns/net ifname=ecs-eth0 containerID=5 bridgeName=ecs-bridge ipamType=ecs-ipam
2025-06-09T07:51:57Z [INFO] msg="Creating veth pair for namespace" netns=/host/proc/3399/ns/net ifname=ecs-eth0 containerID=5 bridgeName=ecs-bridge ipamType=ecs-ipam
2025-06-09T07:51:57Z [INFO] msg="Attaching veth pair to bridge" netns=/host/proc/3399/ns/net ifname=ecs-eth0 containerID=5 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth5
2025-06-09T07:51:57Z [INFO] msg="Running IPAM plugin ADD" netns=/host/proc/3399/ns/net ifname=ecs-eth0 containerID=5 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth5
2025-06-09T07:51:57Z [INFO] msg="Configuring container's interface" netns=/host/proc/3399/ns/net ifname=ecs-eth0 containerID=5 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth5
2025-06-09T07:51:57Z [INFO] msg="Configuring bridge" netns=/host/proc/3399/ns/net ifname=ecs-eth0 containerID=5 bridgeName=ecs-bridge ipamType=ecs-ipam hostVethName=veth5
```
Contributor guide
Research direction
Start by examining the ecs-bridge logging path and the seelog behavior referenced in the issue, focusing on startup when multiple tasks are assigned to a newly joined instance. Reproduce the missing and truncated messages shown in the sample logs, then verify that every expected message is present and complete without log rotation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100