canonical / canonical/microceph
Not capable to set up multiple active MDS
- Dominant language
- Go
- Stars
- 396
- Forks
- 74
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 7
Description
If I'm not mistaken, MDS for CephFS use case is CPU-intenstive and single-threaded. That means if we only have one active MDS in the cluster, we rely on a single CPU core for the entire cluster workload.
https://docs.ceph.com/en/squid/start/hardware-recommendations/#cpu
> CephFS Metadata Servers (MDS) are CPU-intensive. They are single-threaded and perform best with CPUs with a high clock rate (GHz).
In the upstream, there is a set of documentation around multiple active MDS and how to leverage it to distribute the CephFS workload across the services including how to scale out MDS using orchestrator. e.g.
https://docs.ceph.com/en/squid/cephfs/multimds/
> You should configure multiple active MDS daemons when your metadata performance is bottlenecked on the single MDS that runs by default.
...
> Workloads that typically benefit from a larger number of active MDS daemons are those with many clients, perhaps working on many separate directories.
In MicroCeph, there is no config knob or subcommand to launch multiple MDS on a physical host so one active MDS out of 3-node cluster is the only possible configuration as far as I see. The single MDS in MicroCeph is going to be the bottleneck always.
```bash
$ snap list microceph
Name Version Rev Tracking Publisher Notes
microceph 19.2.3+snapcf306793a4 1735 squid/stable canonical✓ in-cohort
$ sudo microceph status
MicroCeph deployment summary:
- mc-1 (192.168.123.165)
Services: mds, mgr, mon, osd
Disks: 1
- mc-2 (192.168.123.191)
Services: mds, mgr, mon, osd
Disks: 1
- mc-3 (192.168.123.201)
Services: mds, mgr, mon, osd
Disks: 1
$ sudo microceph.ceph status
cluster:
id: de9f6e99-4c9a-4a68-995b-1a7f1562dda2
health: HEALTH_OK
services:
mon: 3 daemons, quorum mc-1,mc-3,mc-2 (age 19m)
mgr: mc-1(active, since 20m), standbys: mc-3, mc-2
mds: 1/1 daemons up, 2 standby
osd: 3 osds: 3 up (since 19m), 3 in (since 19m)
data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 30 objects, 584 KiB
usage: 81 MiB used, 96 GiB / 96 GiB avail
pgs: 97 active+clean
$ sudo microceph.ceph fs status
lxd_cephfs - 3 clients
==========
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active mc-2 Reqs: 0 /s 12 15 14 9
POOL TYPE USED AVAIL
lxd_cephfs_meta metadata 108k 30.3G
lxd_cephfs_data data 0 30.3G
STANDBY MDS
mc-1
mc-3
MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)
```
Contributor guide
Research direction
Start with the documented CephFS multiple-active-MDS behavior and reproduce the three-node setup using `microceph status`, `microceph.ceph status`, and `microceph.ceph fs status`. Trace the MicroCeph configuration or command entry point that controls MDS placement, then verify that users can configure multiple active MDS daemons and that `fs status` reports them as active rather than standby.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100