channable / channable/opsqueue

CI keeps failing due to timeout

Closed
#20 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
96
Forks
2
Avg merge
1h 2m
Merged PRs (30d)
2

Description

When i open a PR, i often have to run the job a couple of times before it completes in time.

We've had issues before with our integration tests being flaky and becoming deadlocked indefinitely. #6 introduced the 20 second timeout for integration tests, so they fail more quickly once they become deadlocked.

It's also difficult to see what exactly causes the failure. A first step towards resolving this issue could be to see what we can do to improve the output that we get when a test times out on CI, because at time of writing this is just a massive stack trace with mostly callsites originating from pytest plugins and the like.

Either the time-out is just too short for CI, or we're still getting deadlocks. So far, i've not been able to really reproduce any deadlocks by running the tests locally. We could try to relax that timeout a bit more, but not by too much. As the comment above the timeout configuration states, we're already being quite generous with our time limit.

**Examples**

The last four PRs have all seen at least one failure of the integration tests due to it exceeding the time limit:

- #19
- [failed push](https://channable.semaphoreci.com/workflows/56fef772-2c93-4009-a632-fa1b51fe5aa4)
- #18
- [failed push](https://channable.semaphoreci.com/jobs/ef466f0d-4f1c-4e1c-9e8f-883d47d118c5)
- [failed re-run](https://channable.semaphoreci.com/jobs/c496f290-f42b-46ea-9605-13204f6fcae5)
- #17
- [failed push](https://channable.semaphoreci.com/jobs/ed000520-408d-4222-a674-02833521bdf9)
- #16
- [failed merge by Hoff](https://github.com/channable/opsqueue/pull/16#issuecomment-3376018058)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the integration-test timeout configuration referenced in the issue and inspect the CI jobs linked for PRs #16–#19. Compare the timeout output with local runs to determine whether the failures indicate deadlocks or an insufficient CI limit. Done means the cause is identified and the timeout failures produce actionable output or a justified timeout adjustment.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.