cockroachdb / cockroachdb/cockroach

rttanalysis test failure on ppc64le

Open
#95,051 4 comments 0 reactions 0 assignees View on GitHub
C-bug N-followup O-community T-sql-foundations X-blathers-triaged
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Describe the problem
I built the cockroachdb master on ppc64le and when I executed the test suite, I found that there were test failures for pkg/bench/rttanalysis:rttanalysis_test package.

bazel test //pkg/bench/rttanalysis:rttanalysis_test --verbose_failures
INFO: Build options --//build/toolchains:crdb_test_flag and --test_env have changed, discarding analysis cache.
INFO: Analyzed target //pkg/bench/rttanalysis:rttanalysis_test (1 packages loaded, 23053 targets configured).
INFO: Found 1 test target...
TIMEOUT: //pkg/bench/rttanalysis:rttanalysis_test (Summary) /root/.cache/bazel/_bazel_root/7dff85e8e1542b96d2702ede37c07735/execroot/com_github_cockroachdb_cockroach/bazel-out/ppc-fastbuild/testlogs/pkg/bench/rttanalysis/rttanalysis_test/shard_1_of_16/test.log
Target //pkg/bench/rttanalysis:rttanalysis_test up-to-date:
_bazel/bin/pkg/bench/rttanalysis/rttanalysis_test_/rttanalysis_test
INFO: Elapsed time: 1015.093s, Critical Path: 1004.85s
INFO: 468 processes: 2 internal, 466 processwrapper-sandbox.
INFO: Build completed, 1 test FAILED, 468 total actions
//pkg/bench/rttanalysis:rttanalysis_test TIMEOUT in 1 out of 16 in 900.0s
Stats over 16 runs: max = 900.0s, min = 0.9s, avg = 57.2s, dev = 217.6s /root/.cache/bazel/_bazel_root/7dff85e8e1542b96d2702ede37c07735/execroot/com_github_cockroachdb_cockroach/bazel-out/ppc-fastbuild/testlogs/pkg/bench/rttanalysis/rttanalysis_test/shard_1_of_16/test.log
Executed 1 out of 1 test: 1 fails locally.
INFO: Build completed, 1 test FAILED, 468 total actions

When I analyzed the log got TIMEOUT issue for shard_1_of_16 test.log:
shared the failure log below:

initialized metamorphic constant "span-reuse-rate" with value 4
initialized metamorphic constant "parse-json-impl" with value 1
initialized metamorphic constant "mvcc-incremental-iter-tbi" with value false
initialized metamorphic constant "mvcc-value-disable-simple-encoding" with value true
initialized metamorphic constant "disable-checksstconflicts-range-key-masking" with value true
initialized metamorphic constant "default-batch-bytes-limit" with value 23806
initialized metamorphic constant "kv-batch-size" with value 1
initialized metamorphic constant "row-container-rows-per-chunk-shift" with value 1
initialized metamorphic constant "inverted-joiner-batch-size" with value 1
initialized metamorphic constant "spilling-queue-initial-len" with value 11
initialized metamorphic constant "merge-joiner-groups-buffer" with value 8
initialized metamorphic constant "ColIndexJoin-using-streamer-batch-size" with value 3229477
initialized metamorphic constant "max-batch-size" with value 3641
initialized metamorphic constant "max-batch-byte-size" with value 17040211
initialized metamorphic constant "parallel-scan-result-threshold" with value 5068
initialized metamorphic constant "addsst-rewrite-concurrency" with value 4
initialized metamorphic constant "addsst-rewrite-forced" with value true
initialized metamorphic constant "split-scans-right-for-stats-first" with value true
initialized metamorphic constant "raft-log-truncation-clearrange-threshold" with value 966918
initialized metamorphic constant "span-reuse-rate" with value 50
initialized metamorphic constant "parse-json-impl" with value 0
initialized metamorphic constant "kv.rangefeed.use_dedicated_connection_class.enabled" with value true
initialized metamorphic constant "disable-checksstconflicts-range-key-masking" with value true
initialized metamorphic constant "coldata-batch-size" with value 4063
initialized metamorphic constant "datum-row-converter-batch-size" with value 1
initialized metamorphic constant "row-container-rows-per-chunk-shift" with value 1
initialized metamorphic constant "inverted-joiner-batch-size" with value 1
initialized metamorphic constant "zig-zag-joiner-batch-size" with value 1
initialized metamorphic constant "spilling-queue-initial-len" with value 7
initialized metamorphic constant "merge-joiner-groups-buffer" with value 7
initialized metamorphic constant "ColIndexJoin-using-streamer-batch-size" with value 4795414
initialized metamorphic constant "max-batch-size" with value 1031
initialized metamorphic constant "max-batch-byte-size" with value 28367486
initialized metamorphic constant "parallel-scan-result-threshold" with value 6094
initialized metamorphic constant "copy-batch-size" with value 9668
initialized metamorphic constant "use-index-lookup-for-descriptors-in-database" with value false
initialized metamorphic constant "copy-fast-path-enabled-default" with value false
initialized metamorphic constant "addsst-rewrite-concurrency" with value 10
initialized metamorphic constant "raft-log-truncation-clearrange-threshold" with value 314322
I230111 11:03:53.853771 1 (gostd) rand.go:199 [T1] 1 random seed: 8941451898637608852
I230111 11:03:54.407677 1 (gostd) rand.go:199 [T1] 1 random seed: -6265688189927045670
-- Test timed out at 2023-01-11 11:18:52 UTC --

Environment:

CockroachDB [Master]
Architecture and OS: [ppc64le/RHEL/UBI 8.5]
Go: [1.19.4]
Node: [16.13.0]

Any pointers to resolve the failures would be of great help.

Jira issue: CRDB-23294

Epic CRDB-61004

Contributor guide

Open the contributing guide

Research direction

Start by rerunning Bazel target //pkg/bench/rttanalysis:rttanalysis_test on ppc64le with --verbose_failures and inspect the shard_1_of_16 test.log. Compare the timeout behavior with other architectures and determine what prevents the test from completing within the limit. Done means the rttanalysis test completes successfully on ppc64le without the reported timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.