ComplianceAsCode / ComplianceAsCode/auditree-framework

Add "fast fail" option

Open
#131 0 comments 0 reactions 0 assignees View on GitHub
cli documentation enhancement runners
Dominant language
Python
Stars
76
Forks
25
PR merge metrics
No merged PRs in 30d

Description

## Overview

Adding a fast fail option when fetching and/or checking allows for the stop abruptly when an error is encountered during execution rather than continuing on with processing. This feature allows a user to ensure that reports are generated, fixers executed, remote locker is updated and notifications are sent **ONLY** if all tests (fetchers and checks) have not errored. Failing (not erroring) checks are still valid.

## Requirements

- [ ] Add a "fast fail" option that aborts the current execution immediately (upon first encountered/unexpected error)
- [ ] When "fast fail" is invoked evidence is not committed or pushed to the remote evidence locker
- [ ] When "fast fail" is invoked notifiers will not fire
- [ ] The "fast fail" option will work on a fetcher execution
- [ ] The "fast fail" option will work on a check execution
- [ ] The "fast fail" option will work on a combined fetcher/check execution
- [ ] Current functionality will not be compromised
- [ ] Include documentation for "fast fail"

## Approach

- Add `--fastfail` to CLI options.
- Update runner logic to immediately stop execution when an unexpected error is encountered during a test suite execution.
- Add a fast fail result class for both fetcher and check execution that will stop execution upon first unexpected error.
- Reference: [fetcher test result class](https://github.com/ComplianceAsCode/auditree-framework/blob/main/compliance/runners.py#L365)
- Reference: [check test result class](https://github.com/ComplianceAsCode/auditree-framework/blob/main/compliance/runners.py#L384)
- Leverage [unittest.TextTestResult](https://docs.python.org/3/library/unittest.html#unittest.TestResult) functionality to stop execution upon unexpected error. _One possible route_ for a "fast fail" fetcher test result class would be to set [shouldStop](https://docs.python.org/3/library/unittest.html#unittest.TestResult.shouldStop) to `True` inside of a [stopTest](https://docs.python.org/3/library/unittest.html#unittest.TestResult.stopTest) method when a non-dependency chaining error is encountred during the fetch. Similar can be done for a "fast fail" check test result class although here you would not need to guard against a dependency chaining error.
- Provide different "fast fail" result classes to runners when `--fastfail` option is used.
- [fetcher runner](https://github.com/ComplianceAsCode/auditree-framework/blob/main/compliance/runners.py#L213-L215)
- [check runner](https://github.com/ComplianceAsCode/auditree-framework/blob/main/compliance/runners.py#L332-L334)
- Update CLI execution process to bypass check execution if an error is encountered during a fetcher processing. **This is only applicable when fetchers and checks are executed as part of a single command execution.**
- Update documentation by providing content about the "fast fail" option. Content should include "fast fail" behavior for fetchers, checks and fetchers/checks combined. Provide details on the state of the local locker when execution is aborted due to "fast fail".

## Security and Privacy

N/A

## Test Plan

- Test fetcher only execution
- Test check only execution (If a check errors further processing (reports, fixers, notifiers) will not run)
- Test combined fetcher/check execution
- If a fetcher errors then no checks are to run
- If a check errors further processing (reports, fixers, notifiers) will not run
- Ensure current functionality is not changed when not using the `--fastfail` option

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.