Discussion: Error messages

I've been using SeleniumLibrary for a while now. It's been great. I have come across issues and headaches, but almost all of them are caused by Selenium itself :)

Context

We're using SeleniumLibrary within a significantly sized project with 10+ testers. Some of these testers are not well-versed in code or technical things. Because of this situation, I find myself wrapping most SeleniumLibrary keywords for two reasons: adding retrying behavior and enhancing the error messages. I want to focus on the second reason for this issue.

Our error messages

To me, the error messages are a form of documentation. Documentation that pops up the exact moment you need to know about something. Because of this idea, I write error messages like this:

Element is not visible | Page: 'https://example.com/page' | Element 'div.example.locator' not visible after 5 seconds | Screenshot: 0-FAIL-meaningful-name.png

This error message has the following parts:

1. Element is not visible

   Short description without details. This is useful for quickly scanning multiple fails, especially in our pipeline test report. In the pipeline report, we only have the test name and the fail message. The short message allows you to decide if you want more details. If you do, you keep reading.

2. Page: 'https://example.com/page'

   On what page did the error occur? This is useful for debugging and recognizing known issues. I think known issues should not exist, but reality teaches us that they happen. Being able to recognize them is never a bad thing.

3. Element 'div.example.locator' not visible after 5 seconds

   The actual details of what went wrong. This part can consist of multiple parts, depending on the keyword. The idea is to give as many details as possible. These details are used to figure out what exactly happened. Other examples from our codebase include:

   Simple one:
   Could not scroll element 'div.example.locator' into view

   Multiple parts:
   Element 'div.example.locator' contains unexpected text after 5 seconds | 'Foo' (expected) in 'Who is this Foo fella?' (actual)

   Multiple parts with lots of details:
   Element 'div.example.locator img' attribute 'alt' does not contain 'foo' after 5 seconds | 'Amazing image alt tag contents' does not contain 'foo'

4. Screenshot: 0-FAIL-meaningful-name.png

   The file name of the screenshot. This allows you to easily reference a screenshot if it was made. If no screenshot is made, this part will be Screenshot: [None].

Examples

Same example as above

Element is not visible | Page: 'https://example.com/page' | Element 'div.example.locator' not visible after 5 seconds | Screenshot: 0-FAIL-meaningful-name.png

Lots of details

Element attribute value does not contain expected value | Page: 'https://example.com/page' | Element 'div.example.locator' attribute 'class' does not contain 'foo' after 5 seconds | 'lorum ipsum' does not contain 'foo' | Screenshot: 0-FAIL-meaningful-name.png

Note that the 3rd part is the exact error from Wait until Location contains

Unexpected location | Page: 'https://example.com/page' | Location did not contain 'https://example.com/page' in 5 seconds. | Screenshot: 0-FAIL-meaningful-name.png

Note that the 3rd part is the exact error from Wait Until Element Is Enabled

Element is not enabled | Page: 'https://example.com/page' | Element 'div.example.locator' not enabled in 5 seconds | Screenshot: 0-FAIL-meaningful-name.png

Discussion topic

I'm a fan of solving something like this in a great way, only once, in the right place. I think the right place for these error messages is in SeleniumLibrary. I also think the existing error messages are pretty good but can be improved.

What do you think about adding (some of) these ideas to SeleniumLibrary?

Please note that I'm willing to help implement these changes.