GoogleCloudPlatform / GoogleCloudPlatform/PerfKitBenchmarker

Catch false failures for mysql_service with high thread counts

Open
#1,451 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2k
Forks
562
Avg merge
4h 55m
Merged PRs (30d)
69

Description

[MySQL Storage Benchmark](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/linux_benchmarks/mysql_service_benchmark.py) is reporting failing for thread count 512 even when it is a successful run. This means that completion status is not generated and metrics can not be calculated from thread count 512.

The threads are taking longer to clean up and the data collection continues after the sysbench_run_seconds duration has passed. Below is the error message printed at the bottom of stderr.txt. In this instance the run duration is set to 1200s but data collection continues until 7204s.

Need to implement a more robust way of timing out this command so we don't get false failures and lose results.

Below is the error from stderr:

`(last message repeated 1 times)
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 2013: Lost connection to MySQL server at 'reading initial communication packet', system error: 104
FATAL: unable to connect to MySQL server, aborting...
(last message repeated 1 times)
FATAL: error 2013: Lost connection to MySQL server at 'reading initial communication packet', system error: 104
FATAL: unable to connect to MySQL server, aborting...
FATAL: error 2013: Lost connection to MySQL server at 'reading initial communication packet', system error: 104

STDERR: Warning: Permanently added 'xx.xxx.xxx.xxxx' (ECDSA) to the list of known hosts.
PANIC: unprotected error in call to Lua API (Failed to connect to the database)
PANIC: unprotected error in call to Lua API (Failed to connect to the database)
PANIC: unprotected error in call to Lua API (Failed to connect to the database)

2017-07-17 09:16:07,945 31ae0962 MainThread mysql_service(1/0) INFO Running: ssh -A -p 22 perfkit@xx.xxx.xxx.xxx -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/31ae0962/perfkitbenchmarker_keyfile cat /tmp/pkb/cmd8db59d6e-a620-4527-8959-1140d9675403.log
2017-07-17 09:16:08,744 31ae0962 MainThread mysql_service(1/0) WARNING Exception during RobustRemoteCommand. Wrapper script log:
INFO:root:Acquiring lock on /tmp/pkb/cmd8db59d6e-a620-4527-8959-1140d9675403.status
INFO:root:Started pid 14626: /opt/pkb/bin/sysbench --test=/opt/pkb/share/doc/sysbench/tests/db/oltp.lua --mysql_svc_oltp_tables_count=100 --oltp-table-size=100000 --rand-init=on --db-ps-mode=disable --oltp-dist-type=uniform --oltp-read-only=off --num-threads=512 --percentile=99 --report-interval=2 --max-requests=0 --max-time=1200 --mysql-user=root --mysql-password="xx.xxx.xxx.xxx" --mysql-host=35.192.27.62 run
INFO:root:Waiting on PID 14626
INFO:root:Return code: 1

2017-07-17 09:16:08,744 31ae0962 MainThread mysql_service(1/0) ERROR Error during benchmark mysql_service
Traceback (most recent call last):
File "/usr/local/google/home/PKB/repo3/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 559, in RunBenchmark
DoRunPhase(spec, collector, detailed_timer)
File "/usr/local/google/home/PKB/repo3/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 472, in DoRunPhase
samples = spec.BenchmarkRun(spec)
File "/usr/local/google/home/PKB/repo3/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/mysql_service_benchmark.py", line 930, in Run
results = _RunSysbench(vms[0], metadata)
File "/usr/local/google/home/PKB/repo3/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/mysql_service_benchmark.py", line 372, in _RunSysbench
stdout, _ = _IssueSysbenchCommand(vm, run_seconds, metadata)
File "/usr/local/google/home/PKB/repo3/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/mysql_service_benchmark.py", line 334, in _IssueSysbenchCommand
stdout, stderr = vm.RobustRemoteCommand(run_cmd)
File "/usr/local/google/home/PKB/repo3/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 174, in RobustRemoteCommand
return self.RemoteCommand(' '.join(wait_command), should_log=should_log)
File "/usr/local/google/home/PKB/repo3/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 350, in RemoteCommand
suppress_warning, timeout)
File "/usr/local/google/home/PKB/repo3/PerfKitBenchmarker/perfkitbenchmarker/linux_virtual_machine.py", line 411, in RemoteHostCommand
raise errors.VirtualMachine.RemoteCommandError(error_text)
RemoteCommandError: Got non-zero return code (1) executing python /tmp/pkb/wait_for_command.py --stdout /tmp/pkb/cmd8db59d6e-a620-4527-8959-1140d9675403.stdout --stderr /tmp/pkb/cmd8db59d6e-a620-4527-8959-1140d9675403.stderr --status /tmp/pkb/cmd8db59d6e-a620-4527-8959-1140d9675403.status --delete
Full command: ssh -A -p 22 perfkit@xx.xxx.xxx.xxx -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/31ae0962/perfkitbenchmarker_keyfile python /tmp/pkb/wait_for_command.py --stdout /tmp/pkb/cmd8db59d6e-a620-4527-8959-1140d9675403.stdout --stderr /tmp/pkb/cmd8db59d6e-a620-4527-8959-1140d9675403.stderr --status /tmp/pkb/cmd8db59d6e-a620-4527-8959-1140d9675403.status --delete
STDOUT: sysbench 0.5: multi-threaded system evaluation benchmark
`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.