MarketSquare / MarketSquare/robotframework-retryfailed

Infinite loop of retries for RetryFailed listener

Open
#11 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
19
Forks
8
PR merge metrics
No merged PRs in 30d

Description

I have a test run set up in GCP where in options I included RetryFailed listener and skips for tags "skip":
```
substitutions
...
_ROBOT_OPTIONS: --listener RetryFailed --exclude skip
_HEADLESS: 'True'
...
```

Inside the test case I use tag:
```
[Tags] test:retry(1)
```

I faced an issue that test run is stuck in an infinite loop. In the logs I can see messages:

![retry_loop](https://github.com/user-attachments/assets/a31d8da0-7606-421d-a336-8a59d78fbccc)

...and again, and again: PASS skip FAIL skip. And it retries like this for hours until the job stops by timeout. I have a scheduled run.

I ran the same exact configuration tests run in Docker container locally and RetryFailed listener works as expected, without this infinite loop of retries.

I don't know if anyone stumbled upon this issue, but I found no similar information as to why it might happen.

I'm not sure, but I think the issue might happen because of the thing that in this block of code it sets up the result.status = "SKIP", sets the retry count back to 0, and on the next retry, which shouldn't happen, it loops back again:

```
def end_test(self, test, result):
if self.retries and self._original_log_level is not None:
BuiltIn()._context.output.set_log_level(self._original_log_level)
if not self.max_retries:
self.retries = 0
return
if result.status == "FAIL":
if self.retries < self.max_retries:
index = test.parent.tests.index(test)
test.parent.tests.insert(index + 1, test)
result.status = "SKIP"
result.message += "\nSkipped for Retry"
self.retried_tests.append(test.longname)
self.retries += 1
return
else:
result.message += (
f"{linebreak * bool(result.message)}[RETRY] FAIL on {self.retries}. retry."
)
else:
if self.retries:
result.message += (
f"{linebreak * bool(result.message)}[RETRY] PASS on {self.retries}. retry."
)
self.retries = 0
return
```

The expected behavior is that it should mark test as FAIL after defined number of retries, and then skip to the next test run

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the RetryFailed listener's end_test method shown in the report, then reproduce the _ROBOT_OPTIONS configuration with --listener RetryFailed --exclude skip in GCP and compare it with the local Docker run. Trace the PASS/skip/FAIL/skip sequence and verify that a test is marked FAIL after the configured retries and execution advances to the next test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.