canonical / canonical/cos-proxy-operator

Lots of ARP cache entries

Open
#226 1 comment 0 reactions 0 assignees View on GitHub
Status: Triage Type: Bug
Dominant language
Python
Stars
3
Forks
12
PR merge metrics
No merged PRs in 30d

Description

### Bug Description

While looking at the total amount of arp cache entries per LXD unit in a large environment I found that the `cos-proxy` charm is at the top of the list. By default, on an Ubuntu machine, the arp cache maximum threshold is set to 1024 which can be increased as per the commands shown in https://github.com/canonical/grafana-agent-operator/issues/360. In environments where a single host has plenty of containers the cumulative total amount of arp caches surpasses this default value.

Since metrics are being collected through cos-proxy, when adding new compute nodes we are seeing a proportional increase of arp caches being created. This does make sense. Of course there are other charms that are contributing to the limit as shown below. The purpose of this bug report is two, most importantly its to make this issue visible (add to docs?), and perhaps find a better solution to handle/decrease the amount of entries.

In the log output we can see there is around 200 entries per cos-proxy unit in this 100+ node environment.

### To Reproduce

cos-proxy: 2/stable

### Environment

This is a large OpenStack environment with 100+ nodes.

### Relevant log output

SSH into a machine that hosts the LXD cos-proxy container and run the following to view the amount of ARP entries.

```shell
for i in `sudo lxc list -c n -f csv`; do echo $i ;sudo lxc exec $i -- ip neigh show | wc -l; done
```

The list below (slightly redacted) is the top 15 units I found to have the most ARP entries across 3 machines on which I ran the command above. Please note, the juju unit names was added manually.

```shell
juju--1-lxd-1, cos-proxy/1
210
juju--2-lxd-1, cos-proxy-monitors/1
199
juju--2-lxd-2, rabbitmq-server/1
161
juju--1-lxd-2, rabbitmq-server/0
151
juju--3-lxd-1, rabbitmq-server/2
147
juju--1-lxd-3, ovn-central/0
128
juju--1-lxd-4, placement/0
93
juju--1-lxd-5, neutron-api/0
84
juju--3-lxd-2, ovn-central/2
77
juju--1-lxd-6, cinder/0
73
juju--2-lxd-3, ovn-central/1
71
juju--2-lxd-4, cinder/1
67
juju--3-lxd-3, cinder/2
66
juju--1-lxd-7, mysql-innodb-cluster/0
65
juju--1-lxd-8, masakari/0
55
```

To view the sum of entries in one machines I have provided this example which passes the 1024 default.

```shell
for i in `sudo lxc list -c n -f csv`; do sudo lxc exec $i -- ip neigh show | wc -l; done > sum.txt

awk '{s+=$1} END {print s}' ./sum.txt

1156
```

### Additional context

We ran into this issue because of the following error `neighbour: arp_cache: neighbor table overflow!` in syslog of the host. This cascaded and resulted in network connectivity issues.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported host-level error and reproduce the ARP counts using the provided lxc, ip neigh, and awk commands in a comparable environment. Read the cos-proxy deployment context and determine whether the charm contributes to the overflow; done requires a confirmed cause plus either documented mitigation or a clearly scoped fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python, ubuntu
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.