code-yeongyu / code-yeongyu/senpi
eval: schema advertises language as optional (no description) but parseEvalRequest rejects every omission
- Dominant language
- TypeScript
- Stars
- 429
- Forks
- 98
- Avg merge
- 5h 3m
- Merged PRs (30d)
- 526
Description
### What happened?
In a live session the eval tool rejected every call with `eval run requires language`, leaving the agent with no execution surface (commits, tests, builds all blocked). Transcript forensics: 15 of 58 eval calls failed; every fully-captured failing call sent `{code, summary}` with no `language`, while the successful ones included it. The model kept omitting the field because the advertised schema says that's fine:
`packages/senpi-codemode/src/tool/types.ts` (`createEvalInputSchema`):
```ts
language: Type.Optional(languageSchema), // no description
```
`packages/senpi-codemode/src/tool/eval-request.ts` (`parseEvalRequest`):
```ts
if (!isEvalLanguage(params.language)) throw new TypeError("eval run requires language");
```
So an omitted `language` is schema-valid but dies in `execute()`. The union with `peek`/`stop` presumably can't mark it cleanly required — but unlike `summary` (same required-but-schema-optional situation, resolved via its `"REQUIRED for run..."` description), `language` carries no signal at all. Two aggravators:
- The same TypeError fires for a *missing* and an *invalid* language, so an agent debugging its own call can't tell which happened (mine concluded the harness was dropping its parameters).
- `prepareArguments` in `eval-tool.ts` already repairs/clamps `summary` but does nothing for `language`.
### Steps to reproduce
```ts
import { parseEvalRequest } from "…/senpi-codemode/src/tool/eval-request.ts";
parseEvalRequest({ code: "1", summary: "s" }); // throws, though the schema validates the same params
```
Or run any codemode session and watch a model that omits `language` — every cell fails identically.
### Expected behavior
Maintainer's call, but in increasing size:
1. Minimal, follows the `summary` convention: a description on `languageSchema` ("REQUIRED for run. ...").
2. Split the error so missing vs invalid are distinguishable.
3. Design question rather than assumption: should `prepareArguments` default an omitted `language` to the first enabled one? It would have kept the session alive, but changes semantics (py code with an omitted language would run in the js kernel).
### Version
senpi-codemode 2026.9.5 (npm); verified still present on `main` today.
Contributor guide
Research direction
Start with createEvalInputSchema in packages/senpi-codemode/src/tool/types.ts and parseEvalRequest in packages/senpi-codemode/src/tool/eval-request.ts, then inspect prepareArguments in eval-tool.ts. Reproduce the omitted-language case with parseEvalRequest({ code: "1", summary: "s" }) and compare it with an invalid language; done means the advertised schema and runtime behavior clearly communicate the supported requirement and failure reason.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100