Etcd server apply request took too long
- Dominant language
- Go
- Stars
- 52.3k
- Forks
- 10.5k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 44
Description
### Bug report criteria
- [ ] This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
- [ ] This is not a support request or question, support requests or questions should be raised in the etcd [discussion forums](https://github.com/etcd-io/etcd/discussions).
- [ ] You have read the etcd [bug reporting guidelines](https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/reporting_bugs.md).
- [ ] Existing open issues along with etcd [frequently asked questions](https://etcd.io/docs/latest/faq) have been checked and this is not a duplicate.
### What happened?
We are getting WAL fsync/Backend commit duration alter more frequently.
Instance Size m7g.xlarge
Volume size 10GiB gp3 EBS volume with 4000 IOPS and 750 Throughput
Only one node have issue out of 3. This we are observing almost 5 to 6 Self Managed Ec2 kubernetes cluster.
SignalFX Alert query
A = data('etcd_disk_wal_fsync_duration_seconds_bucket', filter=filter('service.name', 'etcd') and filter('k8s.cluster.name', 'borg-prd-4-aws-ap-southeast-1'), rollup='rate').mean(over='10m').histogram_percentile(pct=99, bucket_name='le').publish(label='WAL fsync P99')
detect(when(A.mean(over='10m') > 0.04, lasting='30m'), auto_resolve_after='900s').publish('P2 WAL fsync duration P99 critical for borg-prd-4-aws-ap-southeast-1')
Dashboard query
A = data('etcd_disk_wal_fsync_duration_seconds_bucket', filter=filter('k8s.cluster.name', 'borg-prd-4-aws-ap-southeast-1') and filter('service.name', 'etcd'), rollup='rate').sum(over='15m').histogram_percentile(pct=99, bucket_name='le').publish(label='WAL fsync')
Logs attached.
[etcd-main.log](https://github.com/user-attachments/files/29499372/etcd-main.log)
[member1.log](https://github.com/user-attachments/files/29499388/member1.log)
[member2.log](https://github.com/user-attachments/files/29499397/member2.log)
### What did you expect to happen?
This latency issue is fixed.
### How can we reproduce it (as minimally and precisely as possible)?
NA
### Anything else we need to know?
Attached Alert and Dashboard snapshot.
### Etcd version (please run commands below)
```console
$ etcd --version
# v3.5.16
$ etcdctl version
# v3
### Etcd configuration (command line flags or environment variables)
# paste your configuration here
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
```console
$ etcdctl member list -w table
# paste output here
$ etcdctl --endpoints= endpoint status -w table
# paste output here
```
### Relevant log output
```Shell
```
Contributor guide
Research direction
Start by reviewing the attached etcd-main.log, member1.log, and member2.log alongside the WAL fsync and backend commit metrics. Compare the affected member with the other two, then determine whether the available configuration and endpoint status data are sufficient to identify the latency cause; done requires a reproducible diagnosis and a validated fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, observability, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100