Layr-Labs / Layr-Labs/d-inference

E2E flake on master: provider WebSocket disconnect under concurrent load → all requests 502

Open
#513 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
527
Forks
114
Avg merge
17h 26m
Merged PRs (30d)
111

Description

## Symptom

`TestIntegration_ConcurrentRequests` and `TestProfile_SingleProviderNonStreaming` fail intermittently on master with:

```
integration_test.go:554: "0" is not greater than "0" — at least some concurrent requests should succeed
status=502 {"code":"provider_error","message":"inference failed after 2 attempt(s): provider disconnected"}
```

The testbed provider drops its WebSocket under concurrent load; retries land on the same (sole) disconnected provider, so every request 502s.

## Evidence

- 4 of the last 5 master runs of `integration.yml` failed with this exact signature (same tests, same assertion line, same error body); only the v0.7.3 run passed.
- Reproduces on master both before (run 28674775656) and after (run 28689461523) the #507 capacity-cooldown merge, so it is not caused by recent coordinator changes.
- Also surfaced on PR #509 (run 28684913472) — confirmed unrelated to that branch's changes (`catalog_size_override.go` / registry tests are not on the failing path).

## Suspects to investigate

- Why the Swift testbed provider's WebSocket drops under concurrent request load (provider-side crash/timeout vs coordinator-side close).
- Single-provider testbed has no failover, so one disconnect fails the whole assertion — consider whether the test should tolerate reconnect-and-retry or assert on the disconnect itself.

Classification from CI investigation: pre-existing master flake, high confidence.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with integration_test.go around line 554 and reproduce the failures from integration.yml, including the cited master runs. Trace the Swift testbed provider WebSocket disconnect under concurrent load and whether the coordinator or provider closes it. Done means identifying the cause and making the concurrent tests stable without masking the provider_error condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, swift
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.