apple / apple/foundationdb

buggified DEGRADED_RESET_INTERVAL can cause test failures

Open
#7,306 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The following PR exposed a simulation failure which was worked around by disabling buggify for DEGRADED_RESET_INTERVAL.
We should revisit whether that can be adjusted or reenabled in the future.

https://github.com/apple/foundationdb/pull/7305

The simulation test discovered a situation where machines attempting to connect
to a dead coordinator (with a well-known endpoint) were getting themselves
marked degraded. This flapping of the degraded state prevented recovery from
completing, as it started over any time it noticed that tlogs on degraded
hosts could be relocated to non-degraded ones.

There is obviously an issue with marking nodes degraded because they keep
trying to connect to a dead coordinator, but that issue is harder to resolve
than the immediate issue of false positive test failures.

Contributor guide

Open the contributing guide

Research direction

Start by reading PR 7305 and the simulation test failure involving DEGRADED_RESET_INTERVAL. Investigate the degraded-state flapping when machines connect to a dead coordinator, then determine whether buggify can be reenabled without causing false-positive simulation failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.