cloud-bulldozer / cloud-bulldozer/benchmark-wrapper

run_snafu does not protect against bad JSON data in certain failures

Open
#372 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19
Forks
56
PR merge metrics
No merged PRs in 30d

Description

Running hammerdb-mssql on kata, we've seen occasional errors indicating parse failure on JSON output. The pod then errors out and there's no way to capture the bad JSON.

IMO it should catch the exception and print a more useful diagnostic, perhaps the bad JSON data.

```
[root@perf-sm5039-3-1 benchmark-runner]# oc logs hammerdb-kata-workload-27917bd1--1-swjfq
2021-11-05T19:16:05Z - INFO - MainProcess - run_snafu: logging level is INFO
2021-11-05T19:16:05Z - INFO - MainProcess - _load_benchmarks: Successfully imported 3 benchmark modules: coremarkpro, systemd_analyze, uperf
2021-11-05T19:16:05Z - INFO - MainProcess - _load_benchmarks: Failed to import 0 benchmark modules:
2021-11-05T19:16:05Z - INFO - MainProcess - run_snafu: Using elasticsearch server with host: http://10.1.184.179:9200
2021-11-05T19:16:05Z - INFO - MainProcess - run_snafu: Using index prefix for ES: hammerdb-test-ci
2021-11-05T19:16:05Z - INFO - MainProcess - run_snafu: Connected to the elasticsearch cluster with info as follows:
/usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py:208: ElasticsearchWarning: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-minimal-setup.html to enable security.
warnings.warn(message, category=ElasticsearchWarning)
2021-11-05T19:16:05Z - INFO - MainProcess - run_snafu: {
"name": "894e68c1af5d",
"cluster_name": "docker-cluster",
"cluster_uuid": "TVGTUkgJStG0xhcZNXGx3w",
"version": {
"number": "7.14.0",
"build_flavor": "default",
"build_type": "docker",
"build_hash": "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
"build_date": "2021-07-29T20:49:32.864135063Z",
"build_snapshot": false,
"lucene_version": "8.9.0",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "You Know, for Search"
}
2021-11-05T19:16:05Z - INFO - MainProcess - py_es_bulk: Using streaming bulk indexer
2021-11-05T19:16:05Z - INFO - MainProcess - wrapper_factory: identified hammerdb as the benchmark wrapper
2021-11-05T19:16:05Z - INFO - MainProcess - trigger_hammerdb: Starting hammerdb run
2021-11-05T19:16:53Z - INFO - MainProcess - trigger_hammerdb: Parsing stdout
2021-11-05T19:16:53Z - INFO - MainProcess - trigger_hammerdb: generating json payload
Traceback (most recent call last):
File "/usr/local/bin/run_snafu", line 33, in
sys.exit(load_entry_point('snafu', 'console_scripts', 'run_snafu')())
File "/opt/snafu/snafu/run_snafu.py", line 142, in main
es, process_generator(index_args, parser), parallel_setting
File "/opt/snafu/snafu/utils/py_es_bulk.py", line 172, in streaming_bulk
for ok, resp_payload in streaming_bulk_generator:
File "/usr/local/lib/python3.6/site-packages/elasticsearch/helpers/actions.py", line 320, in streaming_bulk
actions, chunk_size, max_chunk_bytes, client.transport.serializer
File "/usr/local/lib/python3.6/site-packages/elasticsearch/helpers/actions.py", line 155, in _chunk_actions
for action, data in actions:
File "/opt/snafu/snafu/utils/py_es_bulk.py", line 118, in actions_tracking_closure
for cl_action in cl_actions:
File "/opt/snafu/snafu/run_snafu.py", line 199, in process_generator
for action, index in data_object.emit_actions():
File "/opt/snafu/snafu/hammerdb/trigger_hammerdb.py", line 284, in emit_actions
timestamp,
File "/opt/snafu/snafu/hammerdb/trigger_hammerdb.py", line 178, in _json_payload
"worker": data[i][0],
IndexError: list index out of range
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with snafu/run_snafu.py and snafu/hammerdb/trigger_hammerdb.py, following process_generator into emit_actions and _json_payload where the traceback occurs. Reproduce the malformed JSON or short data failure, then verify that run_snafu reports a useful diagnostic including the problematic data instead of terminating with an unhandled exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.