matrix-org / matrix-org/sytest

tests/50federation/10query-profile.pl flaky "Gateway Time-out"

Open
#362 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Perl
Stars
81
Forks
63
Avg merge
1h 47m
Merged PRs (30d)
1

Description

10query-profile.pl seems to be somewhat unreliable, sometimes fails with

```
Running tests/50federation/10query-profile.pl...
Testing if: Outbound federation can query profile data... FAIL:
| HTTP Request failed (504 Gateway Time-out) at ./run-tests.pl line 542.
+----------------------
```

Moreover, if it fails in this manner it fails *immediately* claiming that timeout, rather than starting up, waiting for a bit, then times out.

Further investigation shows that if you run that one test on its own it seems to quite reliably pass:

```
$ ./run-tests-in-tox --dendron=../dendron/bin/dendron --haproxy tests/50federation/10query-profile.pl
Running tests/50federation/10query-profile.pl...
...
All tests PASSED
```

Moreover, various combinations of tests before it seem to indicate that it is server-startup timing related. By including or omitting various other parts of the test suite before it, if it manages to get to this test file within 1m40 of starting sytest, this test script always passes. If however it hasn't got there before at least 2m06, then this test script always fails.

This also might suggest why this test *seems* to be dendron/haproxy-related, at least on my laptop, because it has always taken longer than this time limit to get there (and hence fails) if running via haproxy+postgres, but gets there much sooner (and hence passes) if running via in-memory sqlite.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/50federation/10query-profile.pl and run it through ./run-tests-in-tox using the dendron and haproxy setup; compare isolated and full-suite runs. Trace startup timing around the 1m40/2m06 threshold and the 504 reported by run-tests.pl line 542. Done means the test no longer flakes in the affected suite while retaining its existing pass behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
perl, postgresql, sqlite
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.