matrix-org / matrix-org/sytest

30rooms/60version_upgrade: "$user_type user has push rules copied to upgraded room" does not properly fail on 0 return

Open
#1,314 2 comments 0 reactions 0 assignees View on GitHub
T-Defect
Dominant language
Perl
Stars
81
Forks
63
Avg merge
1h 47m
Merged PRs (30d)
1

Description

https://github.com/matrix-org/sytest/blob/4784b7ed3337962e7938d368d2748953d627d5bf/tests/30rooms/60version_upgrade.pl#L630-L643

Looking into how tests are returned and checked, it seems that *only* with a `check =>` subroutine it'll actually check if the value returned is truthy or not:

https://github.com/matrix-org/sytest/blob/74f593121bcf66193a90a674b20f30a52f5fcb28/run-tests.pl#L846-L856

Finally, `$f_test` is awaited here:

https://github.com/matrix-org/sytest/blob/74f593121bcf66193a90a674b20f30a52f5fcb28/run-tests.pl#L864-L869

---

As the above test does not have a check function, the result of `0` seems to be ignored.

I ran into this while converting the test to complement, and dendrite did not pass the test, [while it does pass the sytest](https://github.com/matrix-org/dendrite/actions/runs/3397310204/jobs/5649526229#step:7:564).

---

I'm not proficient in perl, and the codebase takes odd twists and turns when defining and returning test results like this, so please someone proficient double-check my research, and see if this bug is indeed real.

Contributor guide

Open the contributing guide

Research direction

Start with tests/30rooms/60version_upgrade.pl lines 630-643, then trace result handling in run-tests.pl lines 846-856 and 864-869. Verify whether a returned 0 is ignored when no check subroutine is present, and compare the behavior with the linked Dendrite test run. Done means the test result is correctly treated as a failure, with the relevant behavior covered by the test suite.

Written by the indexing model from the issue text.

Assessment

Tech stack
perl
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.