Test results storage can be invalid if output callback exception is ignored
- Dominant language
- Python
- Stars
- 722
- Forks
- 237
- Avg merge
- 11h 31m
- Merged PRs (30d)
- 4
Description
I'm extending openHTF to write data to our local database. I have noticed that callback execution failure (our callback creates exception) will be ignored. This could mean that operator sees test as PASS but the result data will not be written into database.
Here is a dirty hack how we mark test as FAIL:
```
diff --git a/openhtf/core/test_descriptor.py b/openhtf/core/test_descriptor.py
--- a/openhtf/core/test_descriptor.py
+++ b/openhtf/core/test_descriptor.py
@@ -343,17 +328,16 @@ class Test(object):
_LOG.debug('Test completed for %s, outputting now.',
final_state.test_record.metadata['test_name'])
for output_cb in self._test_options.output_callbacks:
try:
output_cb(final_state.test_record)
- except Exception: # pylint: disable=broad-except
- stacktrace = traceback.format_exc()
- _LOG.error(
- 'Output callback %s raised:\n%s\nContinuing anyway...',
- output_cb, stacktrace)
-
+ except Exception as _err: # pylint: disable=broad-except
+ _LOG.exception(
+ 'Output callback %s raised', output_cb)
+ final_state.test_record.outcome = test_record.Outcome.FAIL
+ final_state.test_record.outcome_details.append(
+ test_record.OutcomeDetails(
+ 0, 'Output callback %s raised [%s]' % (output_cb,_err)))
# Make sure the final outcome of the test is printed last and in a
# noticeable color so it doesn't get scrolled off the screen or missed.
if final_state.test_record.outcome == test_record.Outcome.ERROR:
```
Contributor guide
Assessment
This issue has not been assessed yet.