canonical / canonical/microceph

OSD/daemon fails to start with "too many elements in the map" on host with many VMs/network interfaces

Open
#669 10 comments 0 reactions 0 assignees View on GitHub
blocked bug
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

### Description
We are experiencing a recurring issue where microceph.osd and microceph.daemon fail to start on one node in a MicroCloud cluster.

The failure is consistent and logged as:
`too many elements in the map: File exists`
This has happened twice, always on the same node. Last time we eventually got it started after two weeks of restart attempts of snap.microceph.osd and snap.microceph.daemon.

### Environment
MicroCeph snap: 19.2.1+snap8295212ceb (squid/stable, in-cohort)
Ubuntu: 24.04
Deployment: MicroCloud (LXD + MicroCeph + MicroOVN)
Hardware: Dell R6515
Node role: compute + storage
Affected node runs ~110 active VMs

### Trigger
Both incidents occurred immediately after an automatic snap refresh of MicroCeph, which triggered a restart of MicroCeph services.
After the restart, MicroCeph failed to come back up on this node.

### Observed behavior
- snap.microceph.osd and snap.microceph.daemon exit immediately with status 1
- All OSDs on the affected host go down, cluster remains up
- Repeated restarts show the same error

Journal:

> microceph.osd[...] : too many elements in the map: File exists
> microceph.daemon[...] : too many elements in the map: File exists

Also
> sudo microceph status
> too many elements in the map: File exists

### Commands tested

The following commands were repeatedly tested without success:
> snap stop microceph
> rm -f /var/snap/microceph/common/var/lock/*
> rm -f /var/snap/microceph/common/run/*.pid
> snap start microceph
>
> systemctl enable --now snap.microceph.osd
> systemctl enable --now snap.microceph.daemon

### Additional observations
The affected host has a very high number of network interfaces:
> ls -1 /sys/class/net | wc -l
> 127
One thought is the failure may be related to netlink / interface enumeration on hosts with many vNICs (as a result of running many VM-instances).

### Expected behavior
MicroCeph services should restart reliably after a snap refresh, even on hosts with many VMs and network interfaces.

### Questions
- Is this a known limitation or bug in MicroCeph?
- Is the too many elements in the map error expected in this scenario?
- Are there recommended mitigations or configuration changes?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.