agentscope-ai / agentscope-ai/AgentTeams
Add task progress watchdog for stuck workers
- Dominant language
- Go
- Stars
- 5.6k
- Forks
- 692
- Avg merge
- 5d 4h
- Merged PRs (30d)
- 23
Description
## Background
Discussion #944 raises a runtime failure mode in Manager/Worker setups: a Worker can stay alive and keep consuming tokens while making no forward progress, for example by repeating the same action or waiting silently. HiClaw already has heartbeat checks, but the current checks are mostly about reachability/readiness and follow-up messages, not structured task-level progress detection.
Related discussion: https://github.com/agentscope-ai/HiClaw/discussions/944
## Problem to avoid
This feature should reduce cases where:
- A Worker is Running/Ready but stuck on an assigned task.
- A Worker repeatedly performs the same action and burns tokens.
- The Manager assumes progress because the Worker is alive or occasionally replies.
- A human admin only notices the problem after a long delay.
- Project/task metadata remains `assigned` or `[~]` even though no meaningful work is happening.
## Current behavior
Manager heartbeat already checks active tasks, ensures the Worker container is ready, asks for progress, and escalates when there is no response for more than one heartbeat cycle. Workers are also instructed to write progress logs under `shared/tasks/{task-id}/progress/YYYY-MM-DD.md`.
The missing piece is that heartbeat does not currently compare progress signals across heartbeat cycles. It cannot reliably distinguish:
- normal long-running work,
- no new progress,
- repeated identical progress/action,
- blocked work,
- runtime/session uncertainty.
## Proposed implementation
Add a lightweight task progress watchdog on top of the existing heartbeat path. This should reuse existing task state and progress logs rather than introducing a new orchestration layer first.
Suggested first version:
1. Extend Manager-side task state for finite active tasks, for example in `state.json.active_tasks[]`:
- `last_progress_at`
- `last_progress_fingerprint`
- `stale_heartbeat_count`
- `last_watchdog_action`
2. Add a deterministic script, for example:
- `manager/agent/skills/task-management/scripts/check-progress-watchdog.sh`
The script should:
- read the task entry from `state.json`,
- inspect the latest progress log under `shared/tasks/{task-id}/progress/`,
- compute a stable fingerprint from the latest meaningful progress block,
- compare it with the previous heartbeat snapshot,
- update `stale_heartbeat_count`,
- output a JSON status such as `normal`, `stale`, `repeated`, `blocked`, or `unknown`.
3. Update `manager/agent/HEARTBEAT.md` so finite task checks call the watchdog before or alongside the existing follow-up message.
4. Escalation policy:
- first stale/repeated cycle: ask the Worker for a concise status/blocker report,
- second consecutive stale/repeated cycle: notify the admin with task id, Worker, last progress summary, and recommended action,
- later cycles: avoid repeatedly pinging the Worker; continue concise admin reporting or recommend intervention/reassignment.
5. Tighten Worker progress-log instructions so progress blocks remain parseable and include enough signal:
- what was done,
- current state,
- issue/blocker if any,
- next step.
## Non-goals for the first version
- Do not rely primarily on `/api/chats` runtime probes for Kubernetes Team Workers until per-Worker service discovery is reliable. Those probes can misreport healthy Workers as unreachable.
- Do not add CRD/status API fields in the first iteration unless needed after the Manager-side watchdog proves useful.
- Do not attempt model-level reasoning over full chat history; use simple deterministic progress-log signals first.
## Acceptance criteria
- Existing heartbeat behavior continues to work for normal active tasks.
- A task with fresh progress does not trigger a stuck warning.
- A task with no new progress across configured heartbeat cycles increments stale state and eventually escalates.
- A task whose latest progress fingerprint repeats across cycles is treated as suspected repeated/no-forward-progress work.
- Completion handling still removes the task from active state.
- The behavior is covered by shell/script tests or a focused integration-style test using fixture task directories and `state.json`.
## Notes
This is intended as an incremental improvement to the existing heartbeat mechanism: not "add heartbeat", but upgrade heartbeat from asking about progress to detecting whether task-level progress is actually changing.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.