etcd-io / etcd-io/etcd

Implement a comprehensive diagnosis tool

Open
#20,217 32 comments 2 reactions 0 assignees View on GitHub
area/etcdctl area/etcdutl area/tooling stage/tracked type/feature
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

### What would you like to be added?

Currently we have several scattered etcd diagnosis tools,
- [etcd-diagnosis](https://github.com/ahrtr/etcd-diagnosis)
- [etcd-dump-logs](https://github.com/etcd-io/etcd/tree/main/tools/etcd-dump-logs)
- [etcd-dump-db](https://github.com/etcd-io/etcd/tree/main/tools/etcd-dump-db)
- [bbolt](https://github.com/etcd-io/bbolt/tree/main/cmd/bbolt)

I think it would be better to deliver a comprehensive tool that integrates all of these capabilities. It should support both online and offline diagnosis:
- online diagnosis connects to a running etcd cluster and gathers diagnostic data.
- offline diagnosis, it analyzes etcd data directly when the etcd instance isn’t running.

A couple of use cases (stories)
- When an user raises an issue, we can request them to use the diagnosis tool to generate a report to gather all required info something like [etcd_diagnosis_report.json](https://github.com/ahrtr/etcd-diagnosis/blob/main/examples/etcd_diagnosis_report.json) to avoid long back and forth communication.
- When the etcd cluster is completely down, and users need to recover the cluster from one of the members using flag `--force-new-cluster`, they need to figure out the best member to restore the cluster from. In such case, they will need to use the diagnosis tool to figure out which member has the latest data.
- When the cluster runs out db space quota, and it's already down. Users will need to figure out which resources consume most of the space. They can use the diagnosis tool.
- Refer to https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/
- For advanced users, they may want to make some offline analysis and gain deeper insights, they can use the diagnosis tool

We can add the comprehensive tool under https://github.com/etcd-io/etcd/tree/main/tools. Once it's done, we can deprecate the existing `etcd-dump-logs` and `etcd-dump-db`

cc @fuweid @ivanvc @jmhbnz @serathius

### Why is this needed?

To improve users diagnosis experience.

### PRs

- Migration of etcd-dump-db
- https://github.com/etcd-io/etcd/pull/20244
- https://github.com/etcd-io/etcd/pull/20248
- Migration of etcd-dump-logs
- tbd...
- Migration of etcd-diagnosis
- https://github.com/etcd-io/etcd/pull/20280

Contributor guide

Open the contributing guide

Research direction

Start by reviewing tools/etcd-dump-logs, tools/etcd-dump-db, the bbolt command, and the referenced etcd-diagnosis report and migration PRs. The issue calls for a comprehensive online and offline diagnosis tool with report generation and analysis capabilities, but does not define a focused first implementation or tests. Check the existing migration work to understand what remains before proposing a scoped contribution.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.