canonical / canonical/cos-proxy-operator
Lots of ARP cache entries
- Dominant language
- Python
- Stars
- 3
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
### Bug Description
While looking at the total amount of arp cache entries per LXD unit in a large environment I found that the `cos-proxy` charm is at the top of the list. By default, on an Ubuntu machine, the arp cache maximum threshold is set to 1024 which can be increased as per the commands shown in https://github.com/canonical/grafana-agent-operator/issues/360. In environments where a single host has plenty of containers the cumulative total amount of arp caches surpasses this default value.
Since metrics are being collected through cos-proxy, when adding new compute nodes we are seeing a proportional increase of arp caches being created. This does make sense. Of course there are other charms that are contributing to the limit as shown below. The purpose of this bug report is two, most importantly its to make this issue visible (add to docs?), and perhaps find a better solution to handle/decrease the amount of entries.
In the log output we can see there is around 200 entries per cos-proxy unit in this 100+ node environment.
### To Reproduce
cos-proxy: 2/stable
### Environment
This is a large OpenStack environment with 100+ nodes.
### Relevant log output
SSH into a machine that hosts the LXD cos-proxy container and run the following to view the amount of ARP entries.
```shell
for i in `sudo lxc list -c n -f csv`; do echo $i ;sudo lxc exec $i -- ip neigh show | wc -l; done
```
The list below (slightly redacted) is the top 15 units I found to have the most ARP entries across 3 machines on which I ran the command above. Please note, the juju unit names was added manually.
```shell
juju--1-lxd-1, cos-proxy/1
210
juju--2-lxd-1, cos-proxy-monitors/1
199
juju--2-lxd-2, rabbitmq-server/1
161
juju--1-lxd-2, rabbitmq-server/0
151
juju--3-lxd-1, rabbitmq-server/2
147
juju--1-lxd-3, ovn-central/0
128
juju--1-lxd-4, placement/0
93
juju--1-lxd-5, neutron-api/0
84
juju--3-lxd-2, ovn-central/2
77
juju--1-lxd-6, cinder/0
73
juju--2-lxd-3, ovn-central/1
71
juju--2-lxd-4, cinder/1
67
juju--3-lxd-3, cinder/2
66
juju--1-lxd-7, mysql-innodb-cluster/0
65
juju--1-lxd-8, masakari/0
55
```
To view the sum of entries in one machines I have provided this example which passes the 1024 default.
```shell
for i in `sudo lxc list -c n -f csv`; do sudo lxc exec $i -- ip neigh show | wc -l; done > sum.txt
awk '{s+=$1} END {print s}' ./sum.txt
1156
```
### Additional context
We ran into this issue because of the following error `neighbour: arp_cache: neighbor table overflow!` in syslog of the host. This cascaded and resulted in network connectivity issues.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reported host-level error and reproduce the ARP counts using the provided lxc, ip neigh, and awk commands in a comparable environment. Read the cos-proxy deployment context and determine whether the charm contributes to the overflow; done requires a confirmed cause plus either documented mitigation or a clearly scoped fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python, ubuntu
- Domain
- infrastructure, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100