canonical / canonical/microceph
OSD/daemon fails to start with "too many elements in the map" on host with many VMs/network interfaces
- Dominant language
- Go
- Stars
- 396
- Forks
- 74
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 7
Description
### Description
We are experiencing a recurring issue where microceph.osd and microceph.daemon fail to start on one node in a MicroCloud cluster.
The failure is consistent and logged as:
`too many elements in the map: File exists`
This has happened twice, always on the same node. Last time we eventually got it started after two weeks of restart attempts of snap.microceph.osd and snap.microceph.daemon.
### Environment
MicroCeph snap: 19.2.1+snap8295212ceb (squid/stable, in-cohort)
Ubuntu: 24.04
Deployment: MicroCloud (LXD + MicroCeph + MicroOVN)
Hardware: Dell R6515
Node role: compute + storage
Affected node runs ~110 active VMs
### Trigger
Both incidents occurred immediately after an automatic snap refresh of MicroCeph, which triggered a restart of MicroCeph services.
After the restart, MicroCeph failed to come back up on this node.
### Observed behavior
- snap.microceph.osd and snap.microceph.daemon exit immediately with status 1
- All OSDs on the affected host go down, cluster remains up
- Repeated restarts show the same error
Journal:
> microceph.osd[...] : too many elements in the map: File exists
> microceph.daemon[...] : too many elements in the map: File exists
Also
> sudo microceph status
> too many elements in the map: File exists
### Commands tested
The following commands were repeatedly tested without success:
> snap stop microceph
> rm -f /var/snap/microceph/common/var/lock/*
> rm -f /var/snap/microceph/common/run/*.pid
> snap start microceph
>
> systemctl enable --now snap.microceph.osd
> systemctl enable --now snap.microceph.daemon
### Additional observations
The affected host has a very high number of network interfaces:
> ls -1 /sys/class/net | wc -l
> 127
One thought is the failure may be related to netlink / interface enumeration on hosts with many vNICs (as a result of running many VM-instances).
### Expected behavior
MicroCeph services should restart reliably after a snap refresh, even on hosts with many VMs and network interfaces.
### Questions
- Is this a known limitation or bug in MicroCeph?
- Is the too many elements in the map error expected in this scenario?
- Are there recommended mitigations or configuration changes?
Contributor guide
Assessment
This issue has not been assessed yet.