apple / apple/foundationdb

Unit tests of GRV requests

Open
#3,869 0 comments 0 reactions 0 assignees View on GitHub
testing
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

When a cluster (in real environment or simulation) does not work as expected, it takes time to identify which component causes the problem.

Since GRV is a critical part in transaction's correctness and performance, we should consider adding unit tests to check its contract.

**Correctness**
- [ ] GRV should monotonically increase, even in different failure scenarios (which will be described later). A test workload can have multiple clients issues GRV and check that the versions monotonically increase per client and across clients;

**Performance**
- [ ] GRV latency should be similar for each client from each proxy;
- [ ] GRV throughput is expected;
- [ ] GRV performance does not degrade much (which will be quantified) when partial failure happens.

**Partial failure**: Failure that does not trigger master recovery.
- Network between a proxy and master or resolver is slinky. The latency on these links is higher;
- A proxy has noisy neighbor and it gets less CPU, cache and memory bandwidth resource;

If only one proxy has the partial failure, an ideal system should redirect traffic to other healthy proxies. The GRV latency should not degrade much. The GRV throughput should only decrease proportional to the number of degraded proxies.

**This is orthogonal to the failure monitoring project**
This issue focuses on testing and understanding if the GRV contract is uphold and how the system's GRV requests reacts to failures.

cc. @sfc-gh-kmakino @sears @yliucode

Contributor guide

Open the contributing guide

Research direction

The issue names no files, existing tests, or specific entry points. Start by locating the GRV request path and test harness, then define workloads for monotonic correctness, client and proxy latency, throughput, and partial failures. Done means repeatable tests quantify the GRV contract and performance behavior under the listed failure scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.