jobizzness / jobizzness/bench

A stalled Devin turn hangs forever: no progress, no timeout, and inbound ACP requests are dropped

Open
#116 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
1h 21m
Merged PRs (30d)
41

Description

## What is wrong

A Devin turn that never starts produces no error, no exit and no timeout. The tab sits on
"working" forever and the only way out is to stop it by hand.

### The evidence

Driving the real `devin acp` with the same handshake `DevinSession` uses
(`initialize` → `authenticate` → `session/new` → `session/prompt`), on a machine where `devin`
had never completed its first-run setup:

- `session/new` succeeded and returned a session id.
- `session/prompt` was accepted.
- One `session_info_update` came back, and then **nothing** for 120s — no
`agent_message_chunk`, no `tool_call`, no error, no process exit. Just the agent's own
`skills discovery` heartbeat every 15s on stderr.

Running `devin -p "Reply with exactly: pong"` once — which printed the first-run banner
("Welcome to Devin CLI! ✓ Logged in as … You're all set.") and then answered — cleared it.
The identical ACP probe then answered in seconds with `stopReason: end_turn`.

So the stall condition is real and reachable: an unfinished first-run state makes the ACP
server accept a prompt and never answer it.

### Why Bench cannot tell

- `src/daemon/devin-session.ts` never emits `progress`. `ClaudeSession` does, and the roster's
liveness (`syncProgress`, `src/daemon/registry.ts:940-955`) is driven by it. A Devin turn with
no tool calls produces no `activity` either, so the row has nothing to update from.
- There is no turn timeout anywhere in `DevinSession`.
- `handle()` (`src/daemon/devin-session.ts:206-262`) ignores **inbound requests** — a JSON-RPC
message with both an `id` and a `method`. `session/request_permission` is a real agent→client
method in the binary's method table. The default session mode is `accept-edits`
(confirmed: `session/new` → `modes.currentModeId === "accept-edits"`), so ordinary file edits
do not ask — but anything that mode does not auto-approve will ask, get no answer, and stall
in exactly the same silent way.

## Acceptance criteria

- [ ] A Devin turn that produces no output for a bounded period surfaces as something the
developer can see and act on, rather than as an indefinite "working". Whether that is a
timeout, a stalled state on the row, or both, is the implementer's call — say which was
chosen and why on this issue.
- [ ] `DevinSession` emits `progress` so the roster's liveness machinery works for it the way it
does for `ClaudeSession`.
- [ ] Inbound requests are answered rather than dropped. At minimum `session/request_permission`
gets a real response; any other inbound request gets a well-formed JSON-RPC error rather
than silence, so the agent fails fast instead of waiting forever.
- [ ] A Devin install that is not ready — first-run setup not completed, not logged in — is
reported when the specialist is created or on its first turn, not discovered as a hang.
`devin doctor --json` exists for this.
- [ ] Tests cover: a fake ACP server that accepts a prompt and goes silent, and one that sends
`session/request_permission`.

## Out of scope

- Deciding Bench's permission policy for Devin, or exposing `session/set_mode`
(Code / Smart / Ask / Plan / Bypass) in the cockpit. Answering the request is in scope;
a UI for choosing how to answer it is not.
- #113, #114, #115.

## Verification

```
pnpm test
pnpm exec tsc -p tsconfig.json --noEmit
```

Manual check a green build will not catch: point `BENCH_DEVIN_BIN` at a stub that accepts
`session/prompt` and never replies, prompt a Devin specialist, and confirm the cockpit says
something true within the timeout instead of sitting on "working".

## Related

- #113, #114, #115 — same file.

Contributor guide

Open the contributing guide

Research direction

Start in src/daemon/devin-session.ts, especially handle() and the turn lifecycle, then compare ClaudeSession progress behavior with registry.ts:940-955. Use the acceptance criteria to define done: silent and unready Devin runs become visible failures, inbound requests receive responses, and fake ACP tests cover both cases; verify with pnpm test and the TypeScript check.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.