canonical / canonical/microceph

Not capable to set up multiple active MDS

Open
#787 3 comments 0 reactions 0 assignees View on GitHub
documentation enhancement
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

If I'm not mistaken, MDS for CephFS use case is CPU-intenstive and single-threaded. That means if we only have one active MDS in the cluster, we rely on a single CPU core for the entire cluster workload.

https://docs.ceph.com/en/squid/start/hardware-recommendations/#cpu
> CephFS Metadata Servers (MDS) are CPU-intensive. They are single-threaded and perform best with CPUs with a high clock rate (GHz).

In the upstream, there is a set of documentation around multiple active MDS and how to leverage it to distribute the CephFS workload across the services including how to scale out MDS using orchestrator. e.g.

https://docs.ceph.com/en/squid/cephfs/multimds/
> You should configure multiple active MDS daemons when your metadata performance is bottlenecked on the single MDS that runs by default.
...
> Workloads that typically benefit from a larger number of active MDS daemons are those with many clients, perhaps working on many separate directories.

In MicroCeph, there is no config knob or subcommand to launch multiple MDS on a physical host so one active MDS out of 3-node cluster is the only possible configuration as far as I see. The single MDS in MicroCeph is going to be the bottleneck always.

```bash
$ snap list microceph
Name Version Rev Tracking Publisher Notes
microceph 19.2.3+snapcf306793a4 1735 squid/stable canonical✓ in-cohort

$ sudo microceph status
MicroCeph deployment summary:
- mc-1 (192.168.123.165)
Services: mds, mgr, mon, osd
Disks: 1
- mc-2 (192.168.123.191)
Services: mds, mgr, mon, osd
Disks: 1
- mc-3 (192.168.123.201)
Services: mds, mgr, mon, osd
Disks: 1

$ sudo microceph.ceph status
cluster:
id: de9f6e99-4c9a-4a68-995b-1a7f1562dda2
health: HEALTH_OK

services:
mon: 3 daemons, quorum mc-1,mc-3,mc-2 (age 19m)
mgr: mc-1(active, since 20m), standbys: mc-3, mc-2
mds: 1/1 daemons up, 2 standby
osd: 3 osds: 3 up (since 19m), 3 in (since 19m)

data:
volumes: 1/1 healthy
pools: 4 pools, 97 pgs
objects: 30 objects, 584 KiB
usage: 81 MiB used, 96 GiB / 96 GiB avail
pgs: 97 active+clean

$ sudo microceph.ceph fs status
lxd_cephfs - 3 clients
==========
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active mc-2 Reqs: 0 /s 12 15 14 9
POOL TYPE USED AVAIL
lxd_cephfs_meta metadata 108k 30.3G
lxd_cephfs_data data 0 30.3G
STANDBY MDS
mc-1
mc-3
MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)
```

Contributor guide

Open the contributing guide

Research direction

Start with the documented CephFS multiple-active-MDS behavior and reproduce the three-node setup using `microceph status`, `microceph.ceph status`, and `microceph.ceph fs status`. Trace the MicroCeph configuration or command entry point that controls MDS placement, then verify that users can configure multiple active MDS daemons and that `fs status` reports them as active rather than standby.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.