aaif-goose / aaif-goose/goose

Scheduled recipe runs that fail mid-stream are recorded and reported as successful

Open
#11,051 7 comments 0 reactions 2 assignees Claimed by @vincenzopalazzo View on GitHub
Dominant language
Rust
Stars
54.2k
Forks
6.2k
Avg merge
3d 2h
Merged PRs (30d)
262

Description

**Describe the bug**

When a scheduled recipe run fails partway through — provider error, rate limit, network drop, anything that surfaces as an `Err` on the agent event stream — the scheduler records and reports it as a successful completion. There is no failure signal anywhere the user can see: the CLI prints success, the job list shows a normal completed run, and the `scheduler_job_failed` telemetry never fires.

`execute_job` breaks out of the stream loop on error and sets a local `stream_error` flag, but that flag is only ever used to pick a label for a tracing counter. The function then returns `Ok(session.id)` on both paths.

`crates/goose/src/scheduler.rs:1123-1141` — the error is captured:

```rust
let mut stream_error = false;
while let Some(message_result) = stream.next().await {
tokio::task::yield_now().await;
match message_result {
Ok(AgentEvent::Message(msg)) => conversation.push(msg),
Ok(AgentEvent::HistoryReplaced(updated)) => conversation = updated,
Ok(_) => {}
Err(e) => {
tracing::error!("Error in agent stream: {}", e);
stream_error = true;
break;
}
}
}
```

`crates/goose/src/scheduler.rs:1154` — its only consumer is a metrics label:

```rust
let exit_type = if stream_error { "error" } else { "normal" };
```

`crates/goose/src/scheduler.rs:1214` — and the function returns success regardless:

```rust
Ok(session.id)
```

Two things follow from that unconditional `Ok`:

1. The cron completion handler at `scheduler.rs:392-396` takes the `Ok(_)` arm, logs `"Job '{id}' completed"`, and skips the `scheduler_job_failed` telemetry emission in the `Err` arm.
2. The `#[cfg(feature = "telemetry")]` block at `scheduler.rs:1191-1210` unconditionally emits `schedule_job_completed` with `"status": "completed"` before returning, so the failed run is affirmatively reported as a success rather than merely going unreported.

`run_now` propagates the same `Ok`, so `goose schedule run-now` prints its success message (`crates/goose-cli/src/commands/schedule.rs:252`) for a run that produced nothing.

---

**To Reproduce**

1. Create a schedule for any recipe.
2. Make the provider fail mid-turn — easiest is to point the configured provider at an unreachable base URL, or use a provider/key combination that returns a rate-limit or auth error after the stream has opened.
3. Trigger the job: `goose schedule run-now --id `, or wait for the cron activation.
4. Observe the CLI reports the run was triggered successfully and returns a session id.
5. `goose schedule sessions --id ` lists the run alongside genuine successes with nothing marking it as failed.
6. The only trace of the failure is an `ERROR Error in agent stream: ...` line in the logs, which nobody is watching for a background job.

---

**Expected behavior**

A run whose event stream terminated on an error should be reported as a failed run:

- `execute_job` should return `Err` (or otherwise propagate a failed outcome) when the stream ended on an error, so the cron handler's existing `Err` arm — error log plus `scheduler_job_failed` telemetry — actually runs.
- The completion telemetry should report a failed status for that run rather than `"completed"`.
- `run_now` should surface the failure to the caller so the CLI does not print a success message.
- The failure should be discoverable after the fact from the schedule's session listing, not only from log scraping.

The partial session should still be persisted — the value of a failed run is being able to inspect how far it got — so this is about the reported outcome, not about discarding work.

---

**Screenshots**

N/A.

---

**Please provide the following information**
- **OS & Arch:** macOS 15 arm64 (code path is platform-independent)
- **Interface:** CLI (and any surface that reads schedule run state)
- **Version:** `main` @ dafdbb736
- **Extensions enabled:** N/A — reproduces with any recipe
- **Provider & Model:** any; the trigger is a mid-stream provider error

---

**Additional context**

This is the same shape as #11047, where a failed ACP turn is reported as a successful `session/prompt` with `stopReason: end_turn`. Scheduled jobs make it worse in practice because there is no human in the loop: a nightly job that has been failing on every activation for a week looks healthy in every surface goose exposes.

While reading this code I also noticed that the cron activation path has no overlap guard — the closure at `scheduler.rs:336` checks only `job.paused`, whereas `run_now` at `scheduler.rs:788` refuses to start with `"Job '{id}' is already running"` when `currently_running` is set. A run that outlasts its own interval therefore runs concurrently with itself, and because both runs write the same key in `running_tasks` (`scheduler.rs:366`, `scheduler.rs:379`) the second run's cancellation token is dropped, which makes `kill_running_job` report success without cancelling anything. That is a distinct bug with a distinct fix; I have kept it out of scope here and am happy to file it separately if it is wanted.

I am not starting implementation — filing this for triage per the contribution workflow.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.