google / google/openhtf

Test results storage can be invalid if output callback exception is ignored

Open
#947 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
722
Forks
237
Avg merge
11h 31m
Merged PRs (30d)
4

Description

I'm extending openHTF to write data to our local database. I have noticed that callback execution failure (our callback creates exception) will be ignored. This could mean that operator sees test as PASS but the result data will not be written into database.

Here is a dirty hack how we mark test as FAIL:
```
diff --git a/openhtf/core/test_descriptor.py b/openhtf/core/test_descriptor.py
--- a/openhtf/core/test_descriptor.py
+++ b/openhtf/core/test_descriptor.py
@@ -343,17 +328,16 @@ class Test(object):

_LOG.debug('Test completed for %s, outputting now.',
final_state.test_record.metadata['test_name'])
for output_cb in self._test_options.output_callbacks:
try:
output_cb(final_state.test_record)
- except Exception: # pylint: disable=broad-except
- stacktrace = traceback.format_exc()
- _LOG.error(
- 'Output callback %s raised:\n%s\nContinuing anyway...',
- output_cb, stacktrace)
-
+ except Exception as _err: # pylint: disable=broad-except
+ _LOG.exception(
+ 'Output callback %s raised', output_cb)
+ final_state.test_record.outcome = test_record.Outcome.FAIL
+ final_state.test_record.outcome_details.append(
+ test_record.OutcomeDetails(
+ 0, 'Output callback %s raised [%s]' % (output_cb,_err)))
# Make sure the final outcome of the test is printed last and in a
# noticeable color so it doesn't get scrolled off the screen or missed.
if final_state.test_record.outcome == test_record.Outcome.ERROR:
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.