canonical / canonical/microceph

Make the cluster-join "Ready dqlite" timeout configurable (joins fail when the dqlite DB is large)

Open
#754 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
396
Forks
74
Avg merge
2d 20h
Merged PRs (30d)
7

Description

### Summary

`microceph cluster join` fails with `Error: Failed to join cluster: Ready dqlite: context deadline exceeded` on clusters whose microcluster **dqlite database has grown large**. The joining node cannot download + replay the dqlite state and reach "ready" within the framework's **hardcoded** ready timeout, so the join times out and rolls back.

**Request:** expose a way to **increase the join/ready timeout** (CLI flag, env var, or config) for clusters with a large dqlite DB and/or slow disks — and, ideally, also let operators **bound the DB size** via dqlite snapshot params (see "Related" below).

### Environment

- MicroCeph **19.2.3** (snap rev **1701**, `squid/stable`), Ceph 19.2.3 Squid
- Ubuntu 24.04, kernel 6.8
- Existing 3-node cluster (mon+osd), adding a **4th** storage node
- 1 GbE management network; OSDs on SATA SSDs

### What happens

On the joining node:

$ sudo microceph cluster join --microceph-ip
Error: Failed to join cluster: Ready dqlite: context deadline exceeded

Joining-node daemon log (`snap.microceph.daemon`): `PreInit` → ~31s of silence → `PreRemove (force=true)` (rollback).

Leader-side daemon log during the attempt:

level=error msg="Received error sending heartbeat to cluster member"
error="Database is still starting" target=":7443"
level=warning msg="Failed to get status of cluster member ... /core/1.0/ready ... connect: connection refused"

The joiner's dqlite never finishes starting within the window.

### Root cause

The microcluster **dqlite DB is ~126 MB** on every member (raft log segments):

$ sudo du -sh /var/snap/microceph/common/state/database/
126M .../database/
# ~25 raft segment files of 4–8 MB each (open-* and -)

On join, the new member must receive + apply this state and reach "ready" within a **hardcoded** deadline:

- `microcluster/internal/db/db.go` wraps the ready-wait in a fixed `context.WithTimeout(...)` (currently `120*time.Second` on `main`; the version vendored in MicroCeph 19.2.3 behaves as ~30s in our testing — PreInit→rollback in ~31s). There is **no flag/env/config** to change it.

The DB is large because microcluster never sets dqlite snapshot params:

- `microcluster/internal/db/dqlite.go` — both `dqlite.New(...)` calls omit `WithSnapshotParams`, so go-dqlite defaults apply (`threshold=1024`, `trailing=8192`). Canonical's own k8s-dqlite docs call these defaults ["too large for small clusters"](https://documentation.ubuntu.com/canonical-kubernetes/release-1.32/snap/reference/troubleshooting/). MicroCeph writes large config/OSD entries to raft, so 8192 trailing
entries ≈ 126 MB.

### Why there's no workaround today

- The join/ready timeout is a compiled-in constant — not exposed on `microceph cluster join` (`--help` has only `--microceph-ip`, `--debug`, `--verbose`, `--state-dir`), not in `ceph config`, not env-driven.
- The dqlite trailing window is not exposed either (no `WithSnapshotParams`, no `tuning.yaml`).
- Result: a cluster with a legitimately large dqlite DB can never add a node, with no operator-facing remedy.

### Request

1. **Make the join/ready timeout configurable** — e.g. `microceph cluster join --timeout`, a daemon config key, or an env var — so large-DB / slow-disk joins can complete.
2. **(Complementary) expose dqlite snapshot params** (threshold/trailing) so operators can bound the DB size, the same way k8s-dqlite already does via `tuning.yaml`:

snapshot:
trailing: 1024
threshold: 512

(Note k8s-dqlite's caveat: set **both** — setting `trailing` alone forces `threshold=0`, snapshotting every transaction.)

### Related / precedent

- k8s-dqlite solved the DB-size side via `tuning.yaml` (link above).
- canonical/microceph #476, #444, #473 (node-join failures) — same `Ready dqlite: context deadline exceeded` symptom; PR #710 fixed an address-selection variant but not the large-DB/timeout case.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.