agentscope-ai / agentscope-ai/AgentTeams
[Proposal] Recover submitted TeamHarness tasks when the completion wake-up is lost
- Ngôn ngữ chính
- Go
- Star
- 5.6k
- Fork
- 692
- Merge trung bình
- 5 ngày 4 giờ
- Pull request đã merge (30 ngày)
- 23
Mô tả
## Summary
On current `main` (`aa650ccacc2ba6171d1b0b5efd2a49b1472abe5d`), TeamHarness persists a Worker result before the Leader continuation is triggered. `taskflow submit_task` writes `TaskMeta.status = submitted`, updates the matching project node, syncs the task, and returns `notificationNeeded`. However, `notificationNeeded` is only a hint; it does not send a message. The Worker must make a separate Matrix/message tool call.
If the Worker turn or process stops after `submit_task` succeeds but before that follow-up message is sent, the result remains available, but no component re-wakes the Leader. The project can remain at `submitted` even though the result and deliverables are present.
I would like to discuss a narrow first recovery slice for TeamHarness projects: recover **submitted-but-unaccepted** tasks with at-least-once wake-up and idempotent state transitions, while keeping result review and acceptance an explicit Leader action.
## Verified reproduction
I reproduced the persistence/notification gap against the exact commit above through the production `server.call_tool("taskflow", ...)` entry point. The external `mc` process boundary was replaced with a successful test double; the real `submit_task`, task/project writes, task sync path, and `notificationNeeded` construction were exercised.
1. Create and delegate a TeamHarness project task.
2. Let the Worker acknowledge it, write `result.md`, and call `taskflow submit_task`.
3. Confirm that the call returns `ok: true`, the task and project node are `submitted`, task sync succeeds, and the response contains `notificationNeeded`.
4. End execution immediately after the MCP result, without making the separate completion-message tool call.
5. Inspect the persisted state and registered background work.
Observed result (repeated three times):
```json
{
"submit_ok": true,
"synced": true,
"task_status": "submitted",
"project_status": "submitted",
"notificationNeeded": {
"event": "submit_task",
"projectId": "project-1",
"targetRoom": "!task-room:example.test"
},
"durable_or_automatic_continuation": false
}
```
Relevant current behavior:
- [`_notification_needed`](https://github.com/agentscope-ai/AgentTeams/blob/aa650ccacc2ba6171d1b0b5efd2a49b1472abe5d/plugins/teamharness/mcp/server.py#L3157-L3168) explicitly says that it does not send a message.
- [`submit_task`](https://github.com/agentscope-ai/AgentTeams/blob/aa650ccacc2ba6171d1b0b5efd2a49b1472abe5d/plugins/teamharness/mcp/server.py#L4181-L4228) persists the submitted state and returns the notification hint.
- The [task-execution skill](https://github.com/agentscope-ai/AgentTeams/blob/aa650ccacc2ba6171d1b0b5efd2a49b1472abe5d/plugins/teamharness/skills/team/task-execution/SKILL.md#L140-L153) requires a separate completion message after submission.
- The [runtime design](https://github.com/agentscope-ai/AgentTeams/blob/aa650ccacc2ba6171d1b0b5efd2a49b1472abe5d/docs/design/teamharness/project-task-runtime-design.md#L387-L407) intentionally leaves runtime hooks, an external continuation/recovery service, active-task scanning, and automatic wake-up for a later design.
This reproduction does not claim a full Kubernetes/Matrix deployment test. It isolates the specific crash window after state persistence and successful storage sync but before the separate Matrix completion message. Current TeamHarness MCP and QwenPaw adapter paths do not register a submitted-task recovery scanner, so no later component owns that persisted continuation obligation.
## Expected invariant
Once a task submission is durably recorded, a transient failure between state persistence and notification must not leave the project permanently stalled.
Recovery may deliver a wake-up more than once, but retries must be safe: repeated scans or messages must not cause repeated result acceptance, repeated requester reports, or repeated downstream project transitions. The Leader must still inspect the submitted result and explicitly call `accept_task_result`; recovery must not accept model output automatically.
## Proposed first slice
Limit the first implementation to one deterministic state:
- `TaskMeta.status == submitted`; and
- the corresponding project node has not yet reached an accepted, revised, blocked, cancelled, or other terminal decision.
For that state, provide a recoverable completion wake-up that:
- survives the originating Worker/runtime process restart;
- uses a stable submission identity (or an equivalent deduplication contract);
- can retry after transient storage, Matrix, or runtime failures;
- fences retries after the result has been accepted/revised, the task has been cancelled, or the project has become terminal;
- preserves the existing `check_task -> accept_task_result` review boundary;
- makes repeated `accept_task_result` handling idempotent for the same submission.
I am intentionally not prescribing the persistence schema or scheduler placement yet; those depend on the ownership decision below.
## Ownership question
TeamHarness currently owns the canonical `TaskMeta`/`ProjectMeta` semantics and the explicit acceptance boundary. At the same time, open PRs [#1169](https://github.com/agentscope-ai/AgentTeams/pull/1169) and [#1172](https://github.com/agentscope-ai/AgentTeams/pull/1172) are adding Controller-level project reads/writes, object-storage freshness, and best-effort Matrix notifications.
Where should the retry loop run?
1. In the Controller, reading canonical project/task state and issuing the wake-up;
2. Through a runtime-neutral TeamHarness reconciler invoked by a thin Leader-runtime scheduler; or
3. In another existing lifecycle component?
My current assumption is that TeamHarness should continue to define eligibility, terminal-state fencing, and idempotency semantics, while the maintainer-chosen lifecycle component owns scheduling. I would appreciate alignment on this boundary before implementation so the recovery path does not duplicate the TeamHarness state machine inside the Controller or depend on an LLM/prompt heartbeat for reliability.
## Non-goals for the first slice
- Promise-to-action enforcement for tasks that never started ([#794](https://github.com/agentscope-ai/AgentTeams/issues/794)).
- No-forward-progress detection for running Workers ([#947](https://github.com/agentscope-ai/AgentTeams/issues/947)).
- ACK-only suppression or general Matrix message-loop handling ([#804](https://github.com/agentscope-ai/AgentTeams/issues/804), [#160](https://github.com/agentscope-ai/AgentTeams/issues/160)).
- General project query/write APIs ([#1169](https://github.com/agentscope-ai/AgentTeams/pull/1169), [#1172](https://github.com/agentscope-ai/AgentTeams/pull/1172)).
- Automatic result acceptance, semantic result evaluation, or full loop-interruption recovery.
- Reliable delivery of the final requester report; that could reuse the mechanism in a later slice if this design works.
- A general-purpose message broker or a claim of exactly-once delivery across object storage and Matrix.
- Runtimes that do not currently produce TeamHarness Project/Task metadata.
## Acceptance criteria for the first slice
- A fault injected after durable `submit_task` persistence but before the completion message is recovered after restart.
- Recovery eventually wakes the correct Leader with the exact project/task/submission identity.
- Repeated scans, delivery retries, and recovery-component restarts cause at most one logical acceptance/project transition for the same submission.
- The normal completion path and the recovery path can race without corrupting state or duplicating downstream transitions.
- Accepted, revised, blocked, cancelled, and terminal tasks are not re-woken.
- Concurrent submitted tasks remain isolated and are routed to their own project/task context.
- Focused tests cover the persistence/notification crash window, retry, idempotency, terminal-state fencing, concurrency, and restart recovery.
- At least one integration-style test crosses canonical TeamHarness metadata and the Matrix notification boundary.
- The runtime design document records the chosen owner and failure semantics.
## Contribution / coordination
I have a deterministic reproduction and have not started the implementation. If this scope is useful and maintainers confirm the ownership boundary, I am happy to take the first slice. I would start with the minimal submission identity/idempotency contract and fault-injection tests, then submit the recovery loop as a separate reviewable PR if preferred.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.