collective / collective/robotsuite

Non critical tests returning "1" on bin/test

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
12
Forks
5
PR merge metrics
No merged PRs in 30d

Description

I have a complex setup on one add-on that marks some tests as non-critical; all critical tests are passing but the `bin/test` command returns "1":

https://s3-sa-east-1.amazonaws.com/collective-cover/collective/collective.cover/2664/2664.1/parts/test/robot_report.html

I can reproduce the problem locally:

```
# bin/test -t "Test File Tile"
Running collective.cover.testing.collective.cover:Robot tests:
Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
Set up plone.testing.z2.Startup in 0.354 seconds.
Set up plone.app.testing.layers.PloneFixture in 10.051 seconds.
Set up collective.cover.testing.Fixture in 10.206 seconds.
Set up plone.app.robotframework.remote.AutoLoginRemoteLibrary:RobotRemote in 0.009 seconds.
Set up plone.testing.z2.ZServer in 0.502 seconds.
Set up collective.cover.testing.collective.cover:Robot in 0.000 seconds.
Running:
1/1 (100.0%)
==============================================================================
Test File Tile
==============================================================================
Test File Tile | FAIL |
Text 'Edit File Tile' did not disappear in 30 seconds
------------------------------------------------------------------------------
Test File Tile | PASS |
0 critical tests, 0 passed, 0 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output: /home/hvelarde/collective/cover/parts/test/test_file_tile/Test_File_Tile/output.xml

Failure in test Test File Tile (test_file_tile.robot)
Traceback (most recent call last):
File "/home/hvelarde/.buildout/eggs/unittest2-0.5.1-py2.7.egg/unittest2/case.py", line 340, in run
testMethod()
File "/home/hvelarde/.buildout/eggs/robotsuite-1.7.0-py2.7.egg/robotsuite/__init__.py", line 461, in runTest
assert last_status == 'PASS', last_message
AssertionError: Text 'Edit File Tile' did not disappear in 30 seconds

The following test left new threads behind:
Test File Tile (test_file_tile.robot)
New thread(s): [<_DummyThread(Dummy-1, started daemon 139729337575168)>]

Ran 1 tests with 1 failures and 0 errors in 1 minutes 6.727 seconds.
Tearing down left over layers:
Tear down collective.cover.testing.collective.cover:Robot in 0.000 seconds.
Tear down plone.app.robotframework.remote.AutoLoginRemoteLibrary:RobotRemote in 0.005 seconds.
Tear down plone.testing.z2.ZServer in 5.077 seconds.
Tear down collective.cover.testing.Fixture in 0.009 seconds.
Tear down plone.app.testing.layers.PloneFixture in 0.085 seconds.
Tear down plone.testing.z2.Startup in 0.006 seconds.
Tear down plone.testing.zca.LayerCleanup in 0.005 seconds.
# echo $?
1
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with bin/test and robotsuite/__init__.py, especially the runTest path shown in the traceback, then reproduce the case using test_file_tile.robot. Trace how critical and non-critical counts are converted into the command exit status. Done means a non-critical failure does not make bin/test return 1, while critical failures still do.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.