buggified DEGRADED_RESET_INTERVAL can cause test failures
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
The following PR exposed a simulation failure which was worked around by disabling buggify for DEGRADED_RESET_INTERVAL.
We should revisit whether that can be adjusted or reenabled in the future.
https://github.com/apple/foundationdb/pull/7305
The simulation test discovered a situation where machines attempting to connect
to a dead coordinator (with a well-known endpoint) were getting themselves
marked degraded. This flapping of the degraded state prevented recovery from
completing, as it started over any time it noticed that tlogs on degraded
hosts could be relocated to non-degraded ones.
There is obviously an issue with marking nodes degraded because they keep
trying to connect to a dead coordinator, but that issue is harder to resolve
than the immediate issue of false positive test failures.
Contributor guide
Research direction
Start by reading PR 7305 and the simulation test failure involving DEGRADED_RESET_INTERVAL. Investigate the degraded-state flapping when machines connect to a dead coordinator, then determine whether buggify can be reenabled without causing false-positive simulation failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100