etcd-io / etcd-io/etcd

Flaky `TestWatchVictims`

Open
#22,266 1 comment 0 reactions 1 assignee Claimed by @Hrishi-Baskaran View on GitHub
area/testing type/flake
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

### Which Github Action / Prow Jobs are flaking?

presubmit unit-test-amd64

### Which tests are flaking?

TestWatchVictims

### Github Action / Prow Job link

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/directory/pull-etcd-unit-test-amd64/2083239017670774784

### Reason for failure (if possible)

The `WatchStream`s in the test have a ten second timeout to watch all events. On a busy CI workflow this may not be enough.

Here's me replicating the failure with stress:
```
--- FAIL: TestWatchVictims (10.26s)
logger.go:146: 2026-08-10T19:32:08.271-0700 INFO bbolt Opening db file (/tmp/TestWatchVictims3841191186/001/etcd_backend_test3716069084/database) with mode -rw------- and with options: {Timeout: 0s, NoGrowSync: false, NoFreelistSync: true, PreLoadFreelist: false, FreelistType: , ReadOnly: false, MmapFlags: 8000, InitialMmapSize: 10737418240, PageSize: 0, MaxSize: 0, NoSync: false, OpenFile: 0x0, Mlock: false, Logger: 0x10fd72702900, NoStatistics: false}
logger.go:146: 2026-08-10T19:32:08.272-0700 INFO bbolt Opening bbolt db (/tmp/TestWatchVictims3841191186/001/etcd_backend_test3716069084/database) successfully
logger.go:146: 2026-08-10T19:32:08.272-0700 DEBUG bbolt Starting a new transaction [writable: true]
logger.go:146: 2026-08-10T19:32:08.272-0700 DEBUG bbolt Starting a new transaction [writable: true] successfully
logger.go:146: 2026-08-10T19:32:08.272-0700 DEBUG bbolt Starting a new transaction [writable: false]
logger.go:146: 2026-08-10T19:32:08.272-0700 DEBUG bbolt Starting a new transaction [writable: false] successfully
logger.go:146: 2026-08-10T19:32:08.272-0700 DEBUG bbolt Creating bucket if not exist "key"
logger.go:146: 2026-08-10T19:32:08.272-0700 DEBUG bbolt Creating bucket if not exist "key" successfully
logger.go:146: 2026-08-10T19:32:08.273-0700 DEBUG bbolt Creating bucket if not exist "meta"
logger.go:146: 2026-08-10T19:32:08.273-0700 DEBUG bbolt Creating bucket if not exist "meta" successfully
logger.go:146: 2026-08-10T19:32:08.273-0700 DEBUG bbolt Committing transaction 2
logger.go:146: 2026-08-10T19:32:08.273-0700 DEBUG bbolt Committing transaction 2 successfully
logger.go:146: 2026-08-10T19:32:08.273-0700 DEBUG bbolt Starting a new transaction [writable: true]
logger.go:146: 2026-08-10T19:32:08.273-0700 DEBUG bbolt Starting a new transaction [writable: true] successfully
logger.go:146: 2026-08-10T19:32:08.273-0700 DEBUG bbolt Starting a new transaction [writable: false]
logger.go:146: 2026-08-10T19:32:08.273-0700 DEBUG bbolt Starting a new transaction [writable: false] successfully
logger.go:146: 2026-08-10T19:32:08.273-0700 INFO kvstore restored {"current-rev": 1}
logger.go:146: 2026-08-10T19:32:09.785-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00\x02_\x00\x00\x00\x00\x00\x00\x00\x00"
logger.go:146: 2026-08-10T19:32:09.785-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00\x02_\x00\x00\x00\x00\x00\x00\x00\x00" successfully
logger.go:146: 2026-08-10T19:32:09.786-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00\x06_\x00\x00\x00\x00\x00\x00\x00\x00"
logger.go:146: 2026-08-10T19:32:09.787-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00\x06_\x00\x00\x00\x00\x00\x00\x00\x00" successfully
logger.go:146: 2026-08-10T19:32:09.787-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00\a_\x00\x00\x00\x00\x00\x00\x00\x00"
logger.go:146: 2026-08-10T19:32:09.787-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00\a_\x00\x00\x00\x00\x00\x00\x00\x00" successfully
logger.go:146: 2026-08-10T19:32:09.787-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00\b_\x00\x00\x00\x00\x00\x00\x00\x00"
...
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00>_\x00\x00\x00\x00\x00\x00\x00\x00" successfully
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00?_\x00\x00\x00\x00\x00\x00\x00\x00"
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00?_\x00\x00\x00\x00\x00\x00\x00\x00" successfully
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00@_\x00\x00\x00\x00\x00\x00\x00\x00"
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00@_\x00\x00\x00\x00\x00\x00\x00\x00" successfully
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00A_\x00\x00\x00\x00\x00\x00\x00\x00"
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Putting key "\x00\x00\x00\x00\x00\x00\x00A_\x00\x00\x00\x00\x00\x00\x00\x00" successfully
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Committing transaction 3
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Committing transaction 3 successfully
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Starting a new transaction [writable: true]
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Starting a new transaction [writable: true] successfully
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Starting a new transaction [writable: false]
logger.go:146: 2026-08-10T19:32:09.878-0700 DEBUG bbolt Starting a new transaction [writable: false] successfully
watchable_store_test.go:944: time out
logger.go:146: 2026-08-10T19:32:18.524-0700 DEBUG bbolt Committing transaction 4
logger.go:146: 2026-08-10T19:32:18.524-0700 DEBUG bbolt Committing transaction 4 successfully
FAIL

ERROR: exit status 1
```

### Anything else we need to know?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.