robotframework / robotframework/robotframework

Alternative status when suite setup or suite teardown fail.

Open
#5,346 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11.9k
Forks
2.6k
Avg merge
1d 18h
Merged PRs (30d)
10

Description

Enhancement Request

This is my use case:

  • When my suite setup fails, I would like to mark the tests as skipped rather than failed. And I would like the final suite status to be failed.
  • When my suite teardown fails, I would like to preserve each tests results, rather than have them marked as failed. And I would like the final suite status to be failed.

The current behaviour is as described in the documentation:

2.4.6 Suite setup and teardown
...

If a suite setup fails, all test cases in it and its child test suites are immediately assigned a fail status and they are not actually executed. [...]

[...] If the suite teardown fails, all test cases in the suite are marked failed, regardless of their original execution status. [...]

...

Maybe my request could be implemented as an alternative behaviour that could be selected from the CLI?
For example: --skip-tests-on-suite-setup-failure and --keep-test-status-on-suite-teardown-failure.

It would also be nice to be able to activate from inside the file using tags: robot:skip-tests-on-suite-setup-failure and robot:keep-test-status-on-suite-teardown-failure. Though I am not sure how this would be possible since tags are attached to tests rather than suites.

Current work around

Currently, I have tried to use try/except statement, which kinds of works but comes with limitations (example here). The tests are marked as skipped in case of suite setup failure, but the whole suite is not marked as failed. Same for suite teardown failure, I can preserve the tests results, but not change the suite setup to failed. Moreover, my keywords have limitations: it is not possible to provide arguments to the keywords and tools like RobotCode do not detect my keywords as used when I call them through my custom Run Suite Setup Keywords and Run Suite Teardown Keywords.

Do you have other suggestions on how to implement this in a custom way if the change request is denied? I would prefer avoiding listeners or post-processing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Robot Framework User Guide section on suite setup and teardown, then examine the CLI and tag-based alternatives proposed in the issue. Define how suite setup and teardown failures should affect individual test statuses and the final suite status, including the requested selectable behavior.

Written by the indexing model from the issue text.

Assessment

Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.