HewlettPackard / HewlettPackard/monkeyble
`should_fail: true` on the last task of a `rescue:` block is not recognized, test reported as failed
- Dominant language
- Python
- Stars
- 134
- Forks
- 15
- PR merge metrics
- No merged PRs in 30d
Description
Migrating `hpe.monkeyble` from 1.4.4 to 1.7.1. One of my scenarios that passed on 1.4.4 now fails on 1.7.1 without any change on my end: a task at the end of a `rescue:` block, deliberately failing to propagate the error, marked `should_fail: true`.
## Steps to reproduce
`playbook.yml`:
```yaml
---
- hosts: localhost
connection: local
gather_facts: false
tasks:
- name: "Try something risky"
block:
- name: "This should fail"
ansible.builtin.fail:
msg: "boom"
rescue:
- name: "Log the rescue"
ansible.builtin.debug:
msg: "rescued"
- name: "Raise final error"
ansible.builtin.fail:
msg: "final propagate"
```
`scenarios.yml`:
```yaml
---
monkeyble_scenarios:
ko_rescue_terminal_failure:
name: "MVE - rescue terminal task marked should_fail: true"
tasks_to_test:
- task: "This should fail"
should_fail: true
- task: "Log the rescue"
should_be_skipped: false
- task: "Raise final error"
should_fail: true
```
Run:
```bash
ANSIBLE_CALLBACKS_ENABLED=hpe.monkeyble.monkeyble_callback \
ansible-playbook playbook.yml -e "@scenarios.yml" -e "monkeyble_scenario=ko_rescue_terminal_failure"
echo "exit code: $?"
```
## Actual result
### With `hpe.monkeyble` 1.4.4
Rescue blocks weren't testable yet at this version: the run stops at the first `should_fail: true` match and never reaches the rescue tasks. Scenario reported as passed.
```
PLAY [localhost] ***************************************************************
🐵 Starting Monkeyble callback 1.4.4
monkeyble_scenario: ko_rescue_terminal_failure
Monkeyble scenario: MVE - rescue terminal task marked should_fail: true
TASK [This should fail] ********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "boom"}
🐵 Monkeyble - Task 'This should fail' - expected 'should_fail': True. actual state: True
🐵 Monkeyble - Task 'This should fail' failed as expected
exit code: 0
```
### With `hpe.monkeyble` 1.7.1 (latest)
```
PLAY [localhost] ***************************************************************
🐵 Starting Monkeyble callback 1.7.1
monkeyble_scenario: ko_rescue_terminal_failure
Monkeyble scenario: MVE - rescue terminal task marked should_fail: true
TASK [This should fail] ********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "boom"}
TASK [Log the rescue] **********************************************************
ok: [localhost] => {
"msg": "rescued"
}
🐵 Monkeyble - Task 'Log the rescue' - expected 'should_be_skipped': False. actual state: False
TASK [Raise final error] *******************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "final propagate"}
PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=1 ignored=0
exit code: 2
```
`Raise final error` fails exactly as declared. Monkeyble now lets the rescue section run (better than 1.4.4), but the scenario is still reported as failed (non-zero exit code), even though the task did exactly what the scenario said it would do.
## Expected result
A task inside `rescue:` (or `always:`) marked `should_fail: true` should be treated like any other `should_fail: true` task: scenario passes, exit code 0.
## Affected versions
- Broken: 1.5.0, 1.6.0, 1.7.0, 1.7.1 (reproduced above on 1.7.1, and confirmed by version bisection that the regression already exists at 1.5.0)
- 1.4.4 and earlier: not affected, but not because rescue is handled correctly.
Monkeyble stops the whole run as soon as the first `should_fail: true` task fails (see the 1.4.4 output above: it exits right after "This should fail" and never even reaches "Log the rescue" or "Raise final error"). The bug can't show up simply because the tasks inside `rescue:` are never evaluated by Monkeyble.
## Environment
- `ansible-core`: 2.19.12
- Python: 3.14.7
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the provided playbook.yml and scenarios.yml command with the hpe.monkeyble.monkeyble_callback entry point. Trace how expected failures are handled for tasks in rescue: and always: blocks, using the reported terminal failure as the case to follow. Done means the scenario accepts that failure and exits with code 0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ansible, python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100