cockroachdb / cockroachdb/cockroach

roachtest: ldr/conflict failed

Open
#173,820 9 comments 0 reactions 0 assignees View on GitHub
A-cdc branch-master C-test-failure O-roachtest O-robot P-3 T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

roachtest.ldr/conflict [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21479083?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_Roachtests_RoachtestNightlyGcePrivateBazel/21479083?buildTab=artifacts#/ldr/conflict) on master @ [1199773ab361336b6bc0a5b31840beb7df0642bf](https://github.com/cockroachdb/cockroach/commits/1199773ab361336b6bc0a5b31840beb7df0642bf):

Failed with:

```
(logical_data_replication.go:618).verifyConflictCorrectness: fingerprints differ with mismatched origin timestamps: RowA={col0_7: 'm.auxnb8s.5jlz.z2q9hcd', col0_8: '0wLzJ', col0_9: '01020000c004000000f628781221bafc411f850b8eb2fcc1413d0a172efc0bdec13d0a572a7395cac1b81e1d1b66b1fa41cdcc3c2febd7f641b81ec5dd511dbcc148e18ee74826f04152b84e74dec4da4114aee3818daef4417b1456feb1c2e1411f8567e62ff9f6418fc2f56438878041ae4741024fbf0142ae47b9eb9a2eebc1295ca72c74a00042', col0_10: '22488', col0_0: '[48 49 48 50 48 48 48 48 67 48 48 52 48 48 48 48 48 48 70 54 50 56 55 56 49 50 50 49 66 65 70 67 52 49 49 70 56 53 48 66 56 69 66 50 70 67 67 49 52 49 51 68 48 65 49 55 50 69 70 67 48 66 68 69 67 49 51 68 48 65 53 55 50 65 55 51 57 53 67 65 67 49 66 56 49 69 49 68 49 66 54 54 66 49 70 65 52 49 67 68 67 67 51 67 50 70 69 66 68 55 70 54 52 49 66 56 49 69 67 53 68 68 53 49 49 68 66 67 67 49 52 56 69 49 56 69 69 55 52 56 50 54 70 48 52 49 53 50 66 56 52 69 55 52 68 69 67 52 68 65 52 49 49 52 65 69 69 51 56 49 56 68 65 69 70 52 52 49 55 66 49 52 53 54 70 69 66 49 67 50 69 49 52 49 49 70 56 53 54 55 69 54 50 70 70 57 70 54 52 49 56 70 67 50 70 53 54 52 51 56 56 55 56 48 52 49 65 69 52 55 52 49 48 50 52 70 66 70 48 49 52 50 65 69 52 55 66 57 69 66 57 65 50 69 69 66 67 49 50 57 53 67 65 55 50 67 55 52 65 48 48 48 52 50]', col0_1: '[52 48 51 57 52 49 51 55 54 53]', col0_2: '[77 46 65 117 88 110 98 56 83 46 53 74 76 90 46 90 50 113 57 104 99 68]', col0_3: '[50 54 55 54 51 48 50 48 57 54]', col0_4: '0000-01-01 20:15:38.253619 +0000 UTC', col0_5: '-22488', col0_6: '[54 53 32 121 101 97 114 115 32 51 32 109 111 110 115 32 50 51 52 32 100 97 121 115 32 49 57 58 48 49 58 52 50 46 57 49 55 57 53 53]'} (mvcc=1787579908.003173142,0, origin=0,0) RowB={col0_7: 'gi', col0_8: '0wLzJ', col0_9: '01010000806666e608102adbc16666de405de9efc1e17a7631741f01c2', col0_10: '22488', col0_0: '[48 49 48 49 48 48 48 48 56 48 54 54 54 54 69 54 48 56 49 48 50 65 68 66 67 49 54 54 54 54 68 69 52 48 53 68 69 57 69 70 67 49 69 49 55 65 55 54 51 49 55 52 49 70 48 49 67 50]', col0_1: '[53 49 48 53 48 52 54 50 49]', col0_2: '[71 73]', col0_3: '[50 54 55 54 51 48 50 48 57 54]', col0_4: '0000-01-01 11:17:59.381249 +0000 UTC', col0_5: '-22488', col0_6: '[52 52 32 121 101 97 114 115 32 57 32 109 111 110 115 32 53 55 52 32 100 97 121 115 32 50 48 58 48 51 58 49 50 46 55 57 52 56 52 49]'} (mvcc=1787579908.046896701,0, origin=1787579907.997257340,0)
test artifacts and logs in: /artifacts/ldr/conflict/run_1
```
Cluster Node to Ip Mapping:
| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21479083-1787550046-147-n7cpu4-0001 | | 10.91.96.53 |
| teamcity-21479083-1787550046-147-n7cpu4-0002 | | 10.91.96.92 |
| teamcity-21479083-1787550046-147-n7cpu4-0003 | | 10.91.96.70 |
| teamcity-21479083-1787550046-147-n7cpu4-0004 | | 10.91.96.25 |
| teamcity-21479083-1787550046-147-n7cpu4-0005 | | 10.91.96.157 |
| teamcity-21479083-1787550046-147-n7cpu4-0006 | | 10.91.96.116 |
| teamcity-21479083-1787550046-147-n7cpu4-0007 | | 10.91.96.208 |

Parameters:
- arch=amd64
- cloud=gce
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=false
- metamorphicLeases=default
- metamorphicWriteBuffering=true
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21479083/ldr-conflict/1787578926603/1787580148331)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21479083-1787550046-147-n7cpu4&from_ts=1787578926603&to_ts=1787580148331&live=false&storage=flex_tier)

/cc @cockroachlabs/cdc

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67120

Contributor guide

Open the contributing guide

Research direction

Start with the ldr/conflict artifacts and logs, then inspect logical_data_replication.go around line 618 and the roachtest README. Compare the mismatched fingerprints and origin timestamps to determine the failure's cause. Done means the ldr/conflict roachtest no longer fails and its conflict correctness check passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.