cockroachdb / cockroachdb/cockroach

roachtest: enable follower-reads/mixed-version/survival=region/locality=global/reads=strong test in shared mode

Open
#157,164 1 comment 0 reactions 0 assignees View on GitHub
branch-master C-bug O-roachtest P-3 T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Roach test for follower reads on global tables has been failing for some time for shared tenants. The test expects follower reads to be served from the nearest replica based on latency, but in some cases, requests are routed to a different replica. A code walkthrough suggests this behavior could result from failed health checks, though no such evidence was observed in the logs. Attempts to reproduce the issue were unsuccessful. As next steps, we plan to disable the test for secondary tenants, add additional logging (maybe using a test knob), and re-enable the test once additional logging is available.

The test is temporarily disabled for shared tenants to avoid noise. This ticket tracks re-enabling it once the issue is fixed.

**To Reproduce**

It wasn't always reproducible but the best possible way to reproduce the issue is to run the test concurrently multiple times.

```
COCKROACH_RANDOM_SEED= roachtest run 'follower-reads/mixed-version/survival=region/locality=global/reads=strong' --count 10
```

**Expected behavior**

The request should be routed to the nearest replica based on latency.

**Environment:**

Roach test on master branch.

**Additional context**

Analysis on the previous failures is available here:

- https://github.com/cockroachdb/cockroach/issues/142194#issuecomment-2801710182
- https://github.com/cockroachdb/cockroach/issues/142194#issuecomment-2802173609
- https://github.com/cockroachdb/cockroach/issues/142194#issuecomment-3452619023
- https://github.com/cockroachdb/cockroach/issues/142194#issuecomment-3478121416
- https://github.com/cockroachdb/cockroach/issues/142194#issuecomment-3480832574

Jira issue: CRDB-56389

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.