matrix-org / matrix-org/sytest
30rooms/60version_upgrade: "$user_type user has push rules copied to upgraded room" does not properly fail on 0 return
- Dominant language
- Perl
- Stars
- 81
- Forks
- 63
- Avg merge
- 1h 47m
- Merged PRs (30d)
- 1
Description
https://github.com/matrix-org/sytest/blob/4784b7ed3337962e7938d368d2748953d627d5bf/tests/30rooms/60version_upgrade.pl#L630-L643
Looking into how tests are returned and checked, it seems that *only* with a `check =>` subroutine it'll actually check if the value returned is truthy or not:
https://github.com/matrix-org/sytest/blob/74f593121bcf66193a90a674b20f30a52f5fcb28/run-tests.pl#L846-L856
Finally, `$f_test` is awaited here:
https://github.com/matrix-org/sytest/blob/74f593121bcf66193a90a674b20f30a52f5fcb28/run-tests.pl#L864-L869
---
As the above test does not have a check function, the result of `0` seems to be ignored.
I ran into this while converting the test to complement, and dendrite did not pass the test, [while it does pass the sytest](https://github.com/matrix-org/dendrite/actions/runs/3397310204/jobs/5649526229#step:7:564).
---
I'm not proficient in perl, and the codebase takes odd twists and turns when defining and returning test results like this, so please someone proficient double-check my research, and see if this bug is indeed real.
Contributor guide
Research direction
Start with tests/30rooms/60version_upgrade.pl lines 630-643, then trace result handling in run-tests.pl lines 846-856 and 864-869. Verify whether a returned 0 is ignored when no check subroutine is present, and compare the behavior with the linked Dendrite test run. Done means the test result is correctly treated as a failure, with the relevant behavior covered by the test suite.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- perl
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100