agent-substrate / agent-substrate/substrate
Fix flaky tests: e2e demo suite timeouts and syncer poll timeout
- Vorherrschende Sprache
- Go
- Sterne
- 1.8k
- Forks
- 316
- Ø Merge
- 2 T. 43 Min.
- Gemergte PRs (30 T.)
- 287
Beschreibung
## Summary
CI failure analysis (`hack/metrics/ci-failure-analysis.py` on branch [`docs/velocity-metrics`](https://github.com/aditya-shantanu/substrate/tree/docs/velocity-metrics) of the aditya-shantanu fork) identified the following tests failing repeatedly across 40 sampled failed CI runs:
| Test | Failures | Job | Root cause |
|---|---|---|---|
| `TestDurableDirLifecycle` | 6× | e2e-test | 503 from atenet-router — xDS routes not yet propagated when actor reaches STATUS_RUNNING |
| `TestActorLifecycle` | 4× | e2e-test | Same: 503 after resume, `callActor` 30s deadline too tight |
| `TestMultipleDurableDirLifecycle` | 2× | e2e-test | Same |
| `TestSyncer_UpdateWorker_RetryOnVersionConflict` | 2× | run-tests | `wait.PollUntilContextTimeout` 5s deadline too tight on loaded CI runners |
| `TestLoaderConcurrentHandshakes` | 2× | run-tests | Concurrent symlink rotation — exact failure mode unknown, needs investigation |
## Root Cause
### e2e demo tests (demo_test.go)
After `waitForActorStatus` returns `STATUS_RUNNING`, the atenet-router's xDS route for the actor isn't always ready immediately. `callActor` retries for 30 seconds, but on a loaded CI runner 503s can persist longer than that. The xDS route propagation delay was already identified and fixed for the networking ingress tests in #724 — the same race affects the demo lifecycle tests.
### TestSyncer_UpdateWorker_RetryOnVersionConflict
The test asserts via `wait.PollUntilContextTimeout(..., 5*time.Second, ...)`. The syncer's rate-limiting work queue adds backoff between retries; on a loaded runner the 5s window is sometimes exhausted before the second attempt completes.
### TestLoaderConcurrentHandshakes
Failure details unavailable from CI logs (logs were truncated / run was re-triggered). Needs a reproduction to diagnose. Leaving for follow-up.
## Fix
See the linked PR. Changes:
- `internal/e2e/suites/demo/demo_test.go`: `callActor` deadline 30s → 90s; `waitForActorStatus` timeout 60s → 120s
- `cmd/ateapi/internal/controlapi/syncer_test.go`: final assertion poll timeout 5s → 15s
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.