google / google/benchmark

[BUG] 1.6.1: test suite is failing

Open
#1,326 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
10.4k
Forks
1.8k
Avg merge
2d 4h
Merged PRs (30d)
8

Description

**Describe the bug**
test suite is failing in 3 units.

**System**
Which OS, compiler, and compiler version are you using:
- OS: Linux x86/64
- Compiler and version: gcc 11.2.1

**To reproduce**
cmake setup:
```console
-- Cache values
BENCHMARK_BUILD_32_BITS:BOOL=OFF
BENCHMARK_DOWNLOAD_DEPENDENCIES:BOOL=OFF
BENCHMARK_ENABLE_ASSEMBLY_TESTS:BOOL=ON
BENCHMARK_ENABLE_DOXYGEN:BOOL=OFF
BENCHMARK_ENABLE_EXCEPTIONS:BOOL=ON
BENCHMARK_ENABLE_GTEST_TESTS:BOOL=OFF
BENCHMARK_ENABLE_INSTALL:BOOL=ON
BENCHMARK_ENABLE_LIBPFM:BOOL=OFF
BENCHMARK_ENABLE_LTO:BOOL=OFF
BENCHMARK_ENABLE_TESTING:BOOL=ON
BENCHMARK_ENABLE_WERROR:BOOL=ON
BENCHMARK_FORCE_WERROR:BOOL=OFF
BENCHMARK_INSTALL_DOCS:BOOL=ON
BENCHMARK_USE_BUNDLED_GTEST:BOOL=OFF
BENCHMARK_USE_LIBCXX:BOOL=OFF
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
CMAKE_INSTALL_PREFIX:PATH=/usr
LLVM_FILECHECK_EXE:PATH=/usr/bin/FileCheck
```
Test suite result:
```console
+ cd benchmark-1.6.1
+ /usr/bin/make -O -j12 V=1 VERBOSE=1 -C x86_64-redhat-linux-gnu test ARGS=--output-on-failure
make: Entering directory '/home/tkloczko/rpmbuild/BUILD/benchmark-1.6.1/x86_64-redhat-linux-gnu'
Running tests...
/usr/bin/ctest --force-new-ctest-process --output-on-failure
Test project /home/tkloczko/rpmbuild/BUILD/benchmark-1.6.1/x86_64-redhat-linux-gnu
Start 1: benchmark
1/66 Test #1: benchmark .................................. Passed 3.85 sec
Start 2: spec_arg
2/66 Test #2: spec_arg ................................... Passed 0.01 sec
Start 3: benchmark_setup_teardown
3/66 Test #3: benchmark_setup_teardown ................... Passed 0.00 sec
Start 4: filter_simple
4/66 Test #4: filter_simple .............................. Passed 0.01 sec
Start 5: filter_simple_list_only
5/66 Test #5: filter_simple_list_only .................... Passed 0.00 sec
Start 6: filter_simple_negative
6/66 Test #6: filter_simple_negative ..................... Passed 0.00 sec
Start 7: filter_simple_negative_list_only
7/66 Test #7: filter_simple_negative_list_only ........... Passed 0.00 sec
Start 8: filter_suffix
8/66 Test #8: filter_suffix .............................. Passed 0.00 sec
Start 9: filter_suffix_list_only
9/66 Test #9: filter_suffix_list_only .................... Passed 0.00 sec
Start 10: filter_suffix_negative
10/66 Test #10: filter_suffix_negative ..................... Passed 0.00 sec
Start 11: filter_suffix_negative_list_only
11/66 Test #11: filter_suffix_negative_list_only ........... Passed 0.00 sec
Start 12: filter_regex_all
12/66 Test #12: filter_regex_all ........................... Passed 0.00 sec
Start 13: filter_regex_all_list_only
13/66 Test #13: filter_regex_all_list_only ................. Passed 0.00 sec
Start 14: filter_regex_all_negative
14/66 Test #14: filter_regex_all_negative .................. Passed 0.00 sec
Start 15: filter_regex_all_negative_list_only
15/66 Test #15: filter_regex_all_negative_list_only ........ Passed 0.00 sec
Start 16: filter_regex_blank
16/66 Test #16: filter_regex_blank ......................... Passed 0.01 sec
Start 17: filter_regex_blank_list_only
17/66 Test #17: filter_regex_blank_list_only ............... Passed 0.00 sec
Start 18: filter_regex_blank_negative
18/66 Test #18: filter_regex_blank_negative ................ Passed 0.00 sec
Start 19: filter_regex_blank_negative_list_only
19/66 Test #19: filter_regex_blank_negative_list_only ...... Passed 0.00 sec
Start 20: filter_regex_none
20/66 Test #20: filter_regex_none .......................... Passed 0.00 sec
Start 21: filter_regex_none_list_only
21/66 Test #21: filter_regex_none_list_only ................ Passed 0.00 sec
Start 22: filter_regex_none_negative
22/66 Test #22: filter_regex_none_negative ................. Passed 0.00 sec
Start 23: filter_regex_none_negative_list_only
23/66 Test #23: filter_regex_none_negative_list_only ....... Passed 0.00 sec
Start 24: filter_regex_wildcard
24/66 Test #24: filter_regex_wildcard ...................... Passed 0.00 sec
Start 25: filter_regex_wildcard_list_only
25/66 Test #25: filter_regex_wildcard_list_only ............ Passed 0.00 sec
Start 26: filter_regex_wildcard_negative
26/66 Test #26: filter_regex_wildcard_negative ............. Passed 0.00 sec
Start 27: filter_regex_wildcard_negative_list_only
27/66 Test #27: filter_regex_wildcard_negative_list_only ... Passed 0.00 sec
Start 28: filter_regex_begin
28/66 Test #28: filter_regex_begin ......................... Passed 0.00 sec
Start 29: filter_regex_begin_list_only
29/66 Test #29: filter_regex_begin_list_only ............... Passed 0.00 sec
Start 30: filter_regex_begin_negative
30/66 Test #30: filter_regex_begin_negative ................ Passed 0.00 sec
Start 31: filter_regex_begin_negative_list_only
31/66 Test #31: filter_regex_begin_negative_list_only ...... Passed 0.00 sec
Start 32: filter_regex_begin2
32/66 Test #32: filter_regex_begin2 ........................ Passed 0.00 sec
Start 33: filter_regex_begin2_list_only
33/66 Test #33: filter_regex_begin2_list_only .............. Passed 0.00 sec
Start 34: filter_regex_begin2_negative
34/66 Test #34: filter_regex_begin2_negative ............... Passed 0.00 sec
Start 35: filter_regex_begin2_negative_list_only
35/66 Test #35: filter_regex_begin2_negative_list_only ..... Passed 0.00 sec
Start 36: filter_regex_end
36/66 Test #36: filter_regex_end ........................... Passed 0.00 sec
Start 37: filter_regex_end_list_only
37/66 Test #37: filter_regex_end_list_only ................. Passed 0.00 sec
Start 38: filter_regex_end_negative
38/66 Test #38: filter_regex_end_negative .................. Passed 0.00 sec
Start 39: filter_regex_end_negative_list_only
39/66 Test #39: filter_regex_end_negative_list_only ........ Passed 0.00 sec
Start 40: options_benchmarks
40/66 Test #40: options_benchmarks ......................... Passed 2.22 sec
Start 41: basic_benchmark
41/66 Test #41: basic_benchmark ............................ Passed 0.68 sec
Start 42: repetitions_benchmark
42/66 Test #42: repetitions_benchmark ...................... Passed 0.04 sec
Start 43: diagnostics_test
43/66 Test #43: diagnostics_test ........................... Passed 0.04 sec
Start 44: skip_with_error_test
44/66 Test #44: skip_with_error_test ....................... Passed 0.15 sec
Start 45: donotoptimize_test
45/66 Test #45: donotoptimize_test ......................... Passed 0.00 sec
Start 46: fixture_test
46/66 Test #46: fixture_test ............................... Passed 0.03 sec
Start 47: register_benchmark_test
47/66 Test #47: register_benchmark_test .................... Passed 0.00 sec
Start 48: map_test
48/66 Test #48: map_test ................................... Passed 0.29 sec
Start 49: multiple_ranges_test
49/66 Test #49: multiple_ranges_test ....................... Passed 0.34 sec
Start 50: args_product_test
50/66 Test #50: args_product_test .......................... Passed 0.26 sec
Start 51: link_main_test
51/66 Test #51: link_main_test ............................. Passed 0.03 sec
Start 52: reporter_output_test
52/66 Test #52: reporter_output_test ....................... Passed 0.31 sec
Start 53: templated_fixture_test
53/66 Test #53: templated_fixture_test ..................... Passed 0.02 sec
Start 54: user_counters_test
54/66 Test #54: user_counters_test ......................... Passed 0.24 sec
Start 55: perf_counters_test
55/66 Test #55: perf_counters_test ......................... Passed 0.00 sec
Start 56: internal_threading_test
56/66 Test #56: internal_threading_test .................... Passed 1.82 sec
Start 57: report_aggregates_only_test
57/66 Test #57: report_aggregates_only_test ................ Passed 0.01 sec
Start 58: display_aggregates_only_test
58/66 Test #58: display_aggregates_only_test ............... Passed 0.00 sec
Start 59: user_counters_tabular_test
59/66 Test #59: user_counters_tabular_test ................. Passed 0.26 sec
Start 60: user_counters_thousands_test
60/66 Test #60: user_counters_thousands_test ............... Passed 0.03 sec
Start 61: memory_manager_test
61/66 Test #61: memory_manager_test ........................ Passed 0.03 sec
Start 62: cxx03
62/66 Test #62: cxx03 ...................................... Passed 0.21 sec
Start 63: complexity_benchmark
63/66 Test #63: complexity_benchmark ....................... Passed 1.43 sec
Start 64: run_donotoptimize_assembly_test_CHECK
64/66 Test #64: run_donotoptimize_assembly_test_CHECK ......***Failed 0.01 sec
donotoptimize_assembly_test.cc:27:17: error: CHECK-LABEL: expected string not found in input
// CHECK-LABEL: test_with_rvalue:
^
/home/tkloczko/rpmbuild/BUILD/benchmark-1.6.1/x86_64-redhat-linux-gnu/test/donotoptimize_assembly_test.s:1:1: note: scanning from here

^
/home/tkloczko/rpmbuild/BUILD/benchmark-1.6.1/x86_64-redhat-linux-gnu/test/donotoptimize_assembly_test.s:76:33: note: possible intended match here
.annobin_donotoptimize_assembly_test.cc_end:
^

Input file: /home/tkloczko/rpmbuild/BUILD/benchmark-1.6.1/x86_64-redhat-linux-gnu/test/donotoptimize_assembly_test.s
Check file: donotoptimize_assembly_test.cc

-dump-input=help explains the following input dump.

Input was:
<<<<<<
1:
label:27'0 X error: no match found
2:
label:27'0 ~
3:
label:27'0 ~
4:
label:27'0 ~
5:
label:27'0 ~
6:
label:27'0 ~
.
.
.
71:
label:27'0 ~
72:
label:27'0 ~
73:
label:27'0 ~
74:
label:27'0 ~
75:
label:27'0 ~
76: .annobin_donotoptimize_assembly_test.cc_end:
label:27'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
label:27'1 ? possible intended match
77: .annobin_donotoptimize_assembly_test.cc_end.hot:
label:27'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
78: .annobin_donotoptimize_assembly_test.cc_end.unlikely:
label:27'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
79: .annobin_donotoptimize_assembly_test.cc_end.startup:
label:27'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
80: .annobin_donotoptimize_assembly_test.cc_end.exit:
label:27'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
81: 0:
label:27'0 ~~~
.
.
.
>>>>>>

Start 65: run_state_assembly_test_CHECK
65/66 Test #65: run_state_assembly_test_CHECK ..............***Failed 0.00 sec
state_assembly_test.cc:17:17: error: CHECK-LABEL: expected string not found in input
// CHECK-LABEL: test_for_auto_loop:
^
/home/tkloczko/rpmbuild/BUILD/benchmark-1.6.1/x86_64-redhat-linux-gnu/test/state_assembly_test.s:1:1: note: scanning from here

^
/home/tkloczko/rpmbuild/BUILD/benchmark-1.6.1/x86_64-redhat-linux-gnu/test/state_assembly_test.s:77:25: note: possible intended match here
.annobin_state_assembly_test.cc_end.hot:
^

Input file: /home/tkloczko/rpmbuild/BUILD/benchmark-1.6.1/x86_64-redhat-linux-gnu/test/state_assembly_test.s
Check file: state_assembly_test.cc

-dump-input=help explains the following input dump.

Input was:
<<<<<<
1:
label:17'0 X error: no match found
2:
label:17'0 ~
3:
label:17'0 ~
4:
label:17'0 ~
5:
label:17'0 ~
6:
label:17'0 ~
.
.
.
72:
label:17'0 ~
73:
label:17'0 ~
74:
label:17'0 ~
75:
label:17'0 ~
76: .annobin_state_assembly_test.cc_end:
label:17'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
77: .annobin_state_assembly_test.cc_end.hot:
label:17'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
label:17'1 ? possible intended match
78: .annobin_state_assembly_test.cc_end.unlikely:
label:17'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
79: .annobin_state_assembly_test.cc_end.startup:
label:17'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
80: .annobin_state_assembly_test.cc_end.exit:
label:17'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
81: 0:
label:17'0 ~~~
82: 1:
label:17'0 ~~~
.
.
.
>>>>>>

Start 66: run_clobber_memory_assembly_test_CHECK
66/66 Test #66: run_clobber_memory_assembly_test_CHECK .....***Failed 0.00 sec
error: no check strings found with prefix 'CHECK-GNU:'

95% tests passed, 3 tests failed out of 66

Total Test time (real) = 12.44 sec

The following tests FAILED:
64 - run_donotoptimize_assembly_test_CHECK (Failed)
65 - run_state_assembly_test_CHECK (Failed)
66 - run_clobber_memory_assembly_test_CHECK (Failed)
Errors while running CTest
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.