agent-substrate / agent-substrate/substrate

Fix flaky tests: e2e demo suite timeouts and syncer poll timeout

未關閉
#799 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/tests kind/bug
主要語言
Go
星號
1.8k
分支
316
平均合併
2 天 43 分鐘
30 天內合併 PR
287

描述

## Summary

CI failure analysis (`hack/metrics/ci-failure-analysis.py` on branch [`docs/velocity-metrics`](https://github.com/aditya-shantanu/substrate/tree/docs/velocity-metrics) of the aditya-shantanu fork) identified the following tests failing repeatedly across 40 sampled failed CI runs:

| Test | Failures | Job | Root cause |
|---|---|---|---|
| `TestDurableDirLifecycle` | 6× | e2e-test | 503 from atenet-router — xDS routes not yet propagated when actor reaches STATUS_RUNNING |
| `TestActorLifecycle` | 4× | e2e-test | Same: 503 after resume, `callActor` 30s deadline too tight |
| `TestMultipleDurableDirLifecycle` | 2× | e2e-test | Same |
| `TestSyncer_UpdateWorker_RetryOnVersionConflict` | 2× | run-tests | `wait.PollUntilContextTimeout` 5s deadline too tight on loaded CI runners |
| `TestLoaderConcurrentHandshakes` | 2× | run-tests | Concurrent symlink rotation — exact failure mode unknown, needs investigation |

## Root Cause

### e2e demo tests (demo_test.go)

After `waitForActorStatus` returns `STATUS_RUNNING`, the atenet-router's xDS route for the actor isn't always ready immediately. `callActor` retries for 30 seconds, but on a loaded CI runner 503s can persist longer than that. The xDS route propagation delay was already identified and fixed for the networking ingress tests in #724 — the same race affects the demo lifecycle tests.

### TestSyncer_UpdateWorker_RetryOnVersionConflict

The test asserts via `wait.PollUntilContextTimeout(..., 5*time.Second, ...)`. The syncer's rate-limiting work queue adds backoff between retries; on a loaded runner the 5s window is sometimes exhausted before the second attempt completes.

### TestLoaderConcurrentHandshakes

Failure details unavailable from CI logs (logs were truncated / run was re-triggered). Needs a reproduction to diagnose. Leaving for follow-up.

## Fix

See the linked PR. Changes:
- `internal/e2e/suites/demo/demo_test.go`: `callActor` deadline 30s → 90s; `waitForActorStatus` timeout 60s → 120s
- `cmd/ateapi/internal/controlapi/syncer_test.go`: final assertion poll timeout 5s → 15s

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。