robotframework / robotframework/robotframework

Failure stdout to show where failure happened

Open
#3,669 6 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
11.9k
Forks
2.6k
Avg merge
1d 18h
Merged PRs (30d)
10

Description

WHY
Current normal Robot Framework command line output does not show where a failure happened.
It will only show failed test case name and a failure message.

Especially when executing tests on a continuous integration server or service, such as GH Actions, Travis CI or CircleCI, first thing I will look at is the terminal output related to the specific task that failed.
Showing directly where the failure happened, would speed up debugging and most likely also make it more likely that easy to fix bugs will be get fixed faster.

Logs are only generated at the end of execution and will also require by default a manual download. This is why I believe they should be used mostly in more demanding debugging situations or when reporting a failure happens far away from time when a failure is fixed.

pytest has a very nice and informative way of showing failure output and this many times points directly the root cause of the problem. For instance a change in input data.

WHAT
Let's show where a failure happens directly on the command line output.

HOW
I propose that the failure message that gets printed to stdout should have the following components:

  1. Failure message from the error as it is already shown
  2. File and line number of the step that is failing
  3. The step as it is in the file (possibly some parts truncated if too big)
  4. The step arguments resolved to values (possibly some parts truncated if too big)

A proof of concept of this can be found from https://github.com/robotframework/robotframework/commit/49dede55654d390bd40fd9e98716330e09ffe0ba

Optimal output would in my opinion look something like this:

==============================================================================
Test
==============================================================================
Test.04 Frames
==============================================================================
Test.04 Frames.Deep Frames
==============================================================================
First level                                                           | PASS |
------------------------------------------------------------------------------
Second level                                                          | PASS |
------------------------------------------------------------------------------
Third level                                                           | FAIL |
> test/resources/frame_stuff.robot:15
> Get Text    ${selector}    ==    This is ${part}
E Get Text    id=b >>> id=c >>> id=cc    ==    This is g
E Property innerText `This is c` should be `This is g`
------------------------------------------------------------------------------
Third level from second                                               | PASS |
------------------------------------------------------------------------------
Test.04 Frames.Deep Frames                                            | FAIL |

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the proof-of-concept commit and the example in test/resources/frame_stuff.robot first. Trace how failed steps are rendered in normal command-line output. Done means failed steps show the existing error, source file and line, step text, and resolved arguments, with truncation where needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.