matrix-org / matrix-org/sytest
sytest won't warn you if you ask it to run tests in a file that doesn't exist
Nobody has claimed this yet.
- Dominant language
- Perl
- Stars
- 81
- Forks
- 63
- Avg merge
- 1h 47m
- Merged PRs (30d)
- 1
Description
There's a typo below (filename is `70oublicroomslist.pl`, not `60`)
```
docker run --rm -v /home/dmr/workspace/synapse:/src:ro -v /home/dmr/workspace/sytest/:/sytest -v /tmp/logs:/logs -e POSTGRES=1 -e WORKERS=1 -e MULTI_POSTGRES=1 matrixdotorg/sytest-synapse:testing --torture-replication 50 -s tests/30rooms/60publicroomslist.pl
```
but sytest happily reports no problems:
```
+ echo '+++ Running tests'
+++ Running tests
+ export COVERAGE_PROCESS_START=/src/.coveragerc
+ COVERAGE_PROCESS_START=/src/.coveragerc
+ RUN_TESTS=(perl -I "$SYTEST_LIB" /sytest/run-tests.pl --python=/venv/bin/python --synapse-directory=/src -B "/src/$BLACKLIST" --coverage -O tap --all --work-directory="/work")
+ '[' -n 1 ']'
+ RUN_TESTS+=(-I Synapse::ViaHaproxy --workers)
+ '[' -n '' ']'
+ mkdir -p /logs
+ TEST_STATUS=0
+ pid=107
+ trap 'kill $pid' TERM INT
+ wait 107
+ perl -I /sytest/lib /sytest/run-tests.pl --python=/venv/bin/python --synapse-directory=/src -B /src/sytest-blacklist --coverage -O tap --all --work-directory=/work -I Synapse::ViaHaproxy --workers --torture-replication 2000 -s tests/30rooms/60publicroomslist.pl
WARN: Use of uninitialized value $test_num in concatenation (.) or string at lib/SyTest/Output/TAP.pm line 52.
+ trap - TERM INT
+ '[' 0 -ne 0 ']'
+ echo -e 'run-tests \e[32mPASSED\e[0m'
run-tests PASSED
+ echo '--- Copying assets'
--- Copying assets
+ rsync --ignore-missing-args --min-size=1B -av /work/server-0 /work/server-1 /logs --include '*/' '--include=*.log.*' '--include=*.log' '--exclude=*'
sending incremental file list
server-0/
server-1/
sent 95 bytes received 20 bytes 230.00 bytes/sec
total size is 0 speedup is 0.00
+ '[' -n '' ']'
+ exit 0
```
This is mildly infuriating.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the missing-file invocation with sytest's run-tests.pl, then inspect lib/SyTest/Output/TAP.pm around line 52 and the runner's handling of the selected test path. The issue is done when a nonexistent test file produces a clear warning or failure instead of reporting PASSED; verify the existing command's output and exit status.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, perl
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100