agent-substrate / agent-substrate/substrate

Fix flaky tests: e2e demo suite timeouts and syncer poll timeout

Đang mở
#799 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/tests kind/bug
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

## Summary

CI failure analysis (`hack/metrics/ci-failure-analysis.py` on branch [`docs/velocity-metrics`](https://github.com/aditya-shantanu/substrate/tree/docs/velocity-metrics) of the aditya-shantanu fork) identified the following tests failing repeatedly across 40 sampled failed CI runs:

| Test | Failures | Job | Root cause |
|---|---|---|---|
| `TestDurableDirLifecycle` | 6× | e2e-test | 503 from atenet-router — xDS routes not yet propagated when actor reaches STATUS_RUNNING |
| `TestActorLifecycle` | 4× | e2e-test | Same: 503 after resume, `callActor` 30s deadline too tight |
| `TestMultipleDurableDirLifecycle` | 2× | e2e-test | Same |
| `TestSyncer_UpdateWorker_RetryOnVersionConflict` | 2× | run-tests | `wait.PollUntilContextTimeout` 5s deadline too tight on loaded CI runners |
| `TestLoaderConcurrentHandshakes` | 2× | run-tests | Concurrent symlink rotation — exact failure mode unknown, needs investigation |

## Root Cause

### e2e demo tests (demo_test.go)

After `waitForActorStatus` returns `STATUS_RUNNING`, the atenet-router's xDS route for the actor isn't always ready immediately. `callActor` retries for 30 seconds, but on a loaded CI runner 503s can persist longer than that. The xDS route propagation delay was already identified and fixed for the networking ingress tests in #724 — the same race affects the demo lifecycle tests.

### TestSyncer_UpdateWorker_RetryOnVersionConflict

The test asserts via `wait.PollUntilContextTimeout(..., 5*time.Second, ...)`. The syncer's rate-limiting work queue adds backoff between retries; on a loaded runner the 5s window is sometimes exhausted before the second attempt completes.

### TestLoaderConcurrentHandshakes

Failure details unavailable from CI logs (logs were truncated / run was re-triggered). Needs a reproduction to diagnose. Leaving for follow-up.

## Fix

See the linked PR. Changes:
- `internal/e2e/suites/demo/demo_test.go`: `callActor` deadline 30s → 90s; `waitForActorStatus` timeout 60s → 120s
- `cmd/ateapi/internal/controlapi/syncer_test.go`: final assertion poll timeout 5s → 15s

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.