etcd-io / etcd-io/etcd

Etcd server apply request took too long

Open
#22,013 1 comment 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
3d 1h
Merged PRs (30d)
44

Description

### Bug report criteria

- [ ] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [ ] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [ ] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [ ] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.

### What happened?

We are getting WAL fsync/Backend commit duration alter more frequently.
Instance Size m7g.xlarge
Volume size 10GiB gp3 EBS volume with 4000 IOPS and 750 Throughput
Only one node have issue out of 3. This we are observing almost 5 to 6 Self Managed Ec2 kubernetes cluster.

SignalFX Alert query
A = data('etcd_disk_wal_fsync_duration_seconds_bucket', filter=filter('service.name', 'etcd') and filter('k8s.cluster.name', 'borg-prd-4-aws-ap-southeast-1'), rollup='rate').mean(over='10m').histogram_percentile(pct=99, bucket_name='le').publish(label='WAL fsync P99')
detect(when(A.mean(over='10m') > 0.04, lasting='30m'), auto_resolve_after='900s').publish('P2 WAL fsync duration P99 critical for borg-prd-4-aws-ap-southeast-1')

Dashboard query
A = data('etcd_disk_wal_fsync_duration_seconds_bucket', filter=filter('k8s.cluster.name', 'borg-prd-4-aws-ap-southeast-1') and filter('service.name', 'etcd'), rollup='rate').sum(over='15m').histogram_percentile(pct=99, bucket_name='le').publish(label='WAL fsync')

Logs attached.

[etcd-main.log](https://github.com/user-attachments/files/29499372/etcd-main.log)

[member1.log](https://github.com/user-attachments/files/29499388/member1.log)

[member2.log](https://github.com/user-attachments/files/29499397/member2.log)

### What did you expect to happen?

This latency issue is fixed.

### How can we reproduce it (as minimally and precisely as possible)?

NA

### Anything else we need to know?

Attached Alert and Dashboard snapshot.

Image

Image

### Etcd version (please run commands below)

```console
$ etcd --version
# v3.5.16

$ etcdctl version
# v3

### Etcd configuration (command line flags or environment variables)

# paste your configuration here

### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

```console
$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints= endpoint status -w table
# paste output here
```

### Relevant log output

```Shell

```

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the attached etcd-main.log, member1.log, and member2.log alongside the WAL fsync and backend commit metrics. Compare the affected member with the other two, then determine whether the available configuration and endpoint status data are sufficient to identify the latency cause; done requires a reproducible diagnosis and a validated fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, observability, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.