collective / collective/robotsuite
Non critical tests returning "1" on bin/test
- Dominant language
- Python
- Stars
- 12
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
I have a complex setup on one add-on that marks some tests as non-critical; all critical tests are passing but the `bin/test` command returns "1":
https://s3-sa-east-1.amazonaws.com/collective-cover/collective/collective.cover/2664/2664.1/parts/test/robot_report.html
I can reproduce the problem locally:
```
# bin/test -t "Test File Tile"
Running collective.cover.testing.collective.cover:Robot tests:
Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
Set up plone.testing.z2.Startup in 0.354 seconds.
Set up plone.app.testing.layers.PloneFixture in 10.051 seconds.
Set up collective.cover.testing.Fixture in 10.206 seconds.
Set up plone.app.robotframework.remote.AutoLoginRemoteLibrary:RobotRemote in 0.009 seconds.
Set up plone.testing.z2.ZServer in 0.502 seconds.
Set up collective.cover.testing.collective.cover:Robot in 0.000 seconds.
Running:
1/1 (100.0%)
==============================================================================
Test File Tile
==============================================================================
Test File Tile | FAIL |
Text 'Edit File Tile' did not disappear in 30 seconds
------------------------------------------------------------------------------
Test File Tile | PASS |
0 critical tests, 0 passed, 0 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output: /home/hvelarde/collective/cover/parts/test/test_file_tile/Test_File_Tile/output.xml
Failure in test Test File Tile (test_file_tile.robot)
Traceback (most recent call last):
File "/home/hvelarde/.buildout/eggs/unittest2-0.5.1-py2.7.egg/unittest2/case.py", line 340, in run
testMethod()
File "/home/hvelarde/.buildout/eggs/robotsuite-1.7.0-py2.7.egg/robotsuite/__init__.py", line 461, in runTest
assert last_status == 'PASS', last_message
AssertionError: Text 'Edit File Tile' did not disappear in 30 seconds
The following test left new threads behind:
Test File Tile (test_file_tile.robot)
New thread(s): [<_DummyThread(Dummy-1, started daemon 139729337575168)>]
Ran 1 tests with 1 failures and 0 errors in 1 minutes 6.727 seconds.
Tearing down left over layers:
Tear down collective.cover.testing.collective.cover:Robot in 0.000 seconds.
Tear down plone.app.robotframework.remote.AutoLoginRemoteLibrary:RobotRemote in 0.005 seconds.
Tear down plone.testing.z2.ZServer in 5.077 seconds.
Tear down collective.cover.testing.Fixture in 0.009 seconds.
Tear down plone.app.testing.layers.PloneFixture in 0.085 seconds.
Tear down plone.testing.z2.Startup in 0.006 seconds.
Tear down plone.testing.zca.LayerCleanup in 0.005 seconds.
# echo $?
1
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with bin/test and robotsuite/__init__.py, especially the runTest path shown in the traceback, then reproduce the case using test_file_tile.robot. Trace how critical and non-critical counts are converted into the command exit status. Done means a non-critical failure does not make bin/test return 1, while critical failures still do.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100