code-yeongyu / code-yeongyu/senpi

eval: schema advertises language as optional (no description) but parseEvalRequest rejects every omission

Open Beginner friendly
#1,395 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
429
Forks
98
Avg merge
5h 3m
Merged PRs (30d)
526

Description

### What happened?

In a live session the eval tool rejected every call with `eval run requires language`, leaving the agent with no execution surface (commits, tests, builds all blocked). Transcript forensics: 15 of 58 eval calls failed; every fully-captured failing call sent `{code, summary}` with no `language`, while the successful ones included it. The model kept omitting the field because the advertised schema says that's fine:

`packages/senpi-codemode/src/tool/types.ts` (`createEvalInputSchema`):
```ts
language: Type.Optional(languageSchema), // no description
```

`packages/senpi-codemode/src/tool/eval-request.ts` (`parseEvalRequest`):
```ts
if (!isEvalLanguage(params.language)) throw new TypeError("eval run requires language");
```

So an omitted `language` is schema-valid but dies in `execute()`. The union with `peek`/`stop` presumably can't mark it cleanly required — but unlike `summary` (same required-but-schema-optional situation, resolved via its `"REQUIRED for run..."` description), `language` carries no signal at all. Two aggravators:

- The same TypeError fires for a *missing* and an *invalid* language, so an agent debugging its own call can't tell which happened (mine concluded the harness was dropping its parameters).
- `prepareArguments` in `eval-tool.ts` already repairs/clamps `summary` but does nothing for `language`.

### Steps to reproduce

```ts
import { parseEvalRequest } from "…/senpi-codemode/src/tool/eval-request.ts";
parseEvalRequest({ code: "1", summary: "s" }); // throws, though the schema validates the same params
```
Or run any codemode session and watch a model that omits `language` — every cell fails identically.

### Expected behavior

Maintainer's call, but in increasing size:

1. Minimal, follows the `summary` convention: a description on `languageSchema` ("REQUIRED for run. ...").
2. Split the error so missing vs invalid are distinguishable.
3. Design question rather than assumption: should `prepareArguments` default an omitted `language` to the first enabled one? It would have kept the session alive, but changes semantics (py code with an omitted language would run in the js kernel).

### Version

senpi-codemode 2026.9.5 (npm); verified still present on `main` today.

Contributor guide

Open the contributing guide

Research direction

Start with createEvalInputSchema in packages/senpi-codemode/src/tool/types.ts and parseEvalRequest in packages/senpi-codemode/src/tool/eval-request.ts, then inspect prepareArguments in eval-tool.ts. Reproduce the omitted-language case with parseEvalRequest({ code: "1", summary: "s" }) and compare it with an invalid language; done means the advertised schema and runtime behavior clearly communicate the supported requirement and failure reason.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.