proposal(runtime): benchmark renaming the Bash tool to shell or exec

English

## Problem

Maka exposes its cross-platform command-execution tool to models as `Bash`, but the name does not accurately describe the execution contract.

The implementation runs commands through the selected platform shell:

- POSIX uses `/bin/sh`.
- Windows may use PowerShell 7, Windows PowerShell 5.1, or `cmd.exe`.
- The tool description already has to tell Windows models explicitly not to write Bash syntax.

Evidence:

- [`Bash` is the model-visible tool name](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/runtime/src/shell-tools.ts#L90-L100)
- [POSIX commands use `/bin/sh`](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/runtime/src/shell-detect.ts#L175-L176)
- [Windows may use PowerShell or `cmd.exe`](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/runtime/src/shell-detect.ts#L179-L193)

This creates two concerns:

1. `Bash` is misleading on Windows and is not strictly correct on POSIX either.
2. Tool names can affect model priors, tool selection, argument generation, and recovery behavior. A name that is semantically cleaner for humans is not necessarily better for every model.

The proposal is therefore to benchmark `Bash`, `shell`, and `exec` before deciding whether to rename the tool.

## Decision sought

Run a controlled, reproducible three-arm ablation:

```text
Bash — current baseline
shell — platform-neutral command-shell terminology
exec — concise execution terminology
```

The benchmark should determine whether either proposed name improves or preserves agent performance sufficiently to justify the compatibility and migration cost.

This issue does not pre-decide that a rename should happen.

## Candidate trade-offs

| Candidate | Advantages | Risks |
| --- | --- | --- |
| `Bash` | Existing model familiarity; no migration | Inaccurate across supported shells, especially on Windows |
| `shell` | Describes the actual cross-platform abstraction | May overlap conceptually with provider-native shell capabilities and external tool names |
| `exec` | Short and common in agent tool surfaces | Maka already uses [`exec` for the code-mode orchestration cell](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/runtime/src/ai-sdk-backend.ts#L1257-L1273), so this candidate requires an explicit collision-resolution design |

## Benchmark design

### Controlled variable

The model-visible function name should be the only changed variable:

- identical system prompt;
- identical tool description;
- identical parameter schema;
- identical runtime and permission behavior;
- identical model/provider configuration;
- identical task order and budgets.

Legacy aliases must not be exposed alongside the tested name. Giving the model multiple equivalent command tools would change the tool surface and invalidate the comparison.

### Evaluation suites

Use both targeted and end-to-end evaluation.

#### 1. Tool-routing suite

Include tasks where:

- a shell command is clearly required;
- `Read`, `Glob`, or `Grep` is preferable to shell execution;
- no tool is required;
- a command fails and the model must recover;
- foreground, background, and PTY behavior are exercised;
- the prompt contains words such as “bash”, “shell”, or “execute” without necessarily requiring that tool.

This suite should measure whether the name changes first-action selection or causes unnecessary command execution.

#### 2. Cross-platform command suite

Run equivalent tasks on at least:

- a POSIX environment using `/bin/sh`;
- Windows with the shell selected by Maka.

Measure both tool selection and shell-dialect correctness. The Windows arm is essential because cross-platform accuracy is the main motivation for reconsidering `Bash`.

#### 3. End-to-end agent benchmark

Run a paired end-to-end benchmark using Maka’s existing evaluation infrastructure.

Terminal-Bench can provide continuity with existing results, but it should not be the only suite: Maka’s own report notes that Terminal-Bench 2.1 has limited discriminative headroom for frontier harnesses. A targeted suite is necessary to isolate tool-name effects.

### Model coverage

Include representative supported model families rather than drawing a conclusion from one provider. Pin exact model identifiers or snapshots wherever possible.

The comparison should preserve the same model and provider within every paired trial.

### Metrics

Primary metrics:

- task success rate;
- correct first tool/action rate in the targeted suite.

Secondary metrics:

- invalid or malformed tool calls;
- unnecessary shell invocations;
- shell-dialect errors;
- command repair/retry count;
- total tool calls and turns;
- tokens, latency, and cost;
- permission or sandbox-flow regressions.

### Experimental discipline

- Pre-register the primary metric and non-inferiority threshold before examining results.
- Use paired tasks and repeated trials where provider nondeterminism requires them.
- Randomize or balance arm order.
- Keep raw traces and failure classifications.
- Publish the experiment manifest, aggregate results, and per-task outcomes.
- Distinguish model errors, provider failures, harness failures, and benchmark timeouts.

## Decision rule

Rename the tool only if the selected candidate:

1. is non-inferior on end-to-end task success;
2. does not regress safety, permission, sandbox, or recovery behavior;
3. improves or preserves targeted tool-selection behavior across representative models;
4. has a viable compatibility plan.

If neither candidate clears those conditions, retain `Bash`.

If `exec` performs best, its existing collision with the code-mode `exec` tool must be resolved before it can be selected.

## Compatibility and migration scope

This is not a cosmetic string replacement. `Bash` is currently used as an identity across:

- the model-facing tool catalog;
- permission and sandbox-escalation records;
- persisted runtime events and transcript replay;
- usage aggregation and search;
- CLI and Desktop presentation;
- shell-run overlays and background-task recovery;
- bundled skills, prompts, tests, and documentation.

For example:

- [sandbox permission records require `toolName === 'Bash'`](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/core/src/interaction-record-schema.ts#L145-L155)
- [the UI recognizes shell-run updates through the exact tool name](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/ui/src/materialize.ts#L573-L584)
- [`Bash` is part of the canonical product tool catalog](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/core/src/tool-catalog.ts#L78-L88)

If a rename is approved, implementation should:

- expose only the new canonical name to models;
- continue reading historical `Bash` events and transcripts;
- preserve permission and sandbox semantics;
- maintain usage/search continuity across old and new names;
- avoid scattering dual-name checks throughout the codebase;
- migrate internal shell recognition toward semantic metadata or one canonicalization boundary;
- define compatibility expectations for external integrations and imported sessions.

The implementation and migration should be handled in a separate issue or PR after the benchmark decision.

## Deliverables

- [ ] Reproducible three-arm benchmark manifest
- [ ] Targeted routing and cross-platform task suite
- [ ] Representative multi-model results
- [ ] Per-task outcomes and failure classifications
- [ ] Recommendation: keep `Bash`, rename to `shell`, or rename to `exec`
- [ ] Compatibility design for the recommended option
- [ ] Follow-up implementation issue if a rename is approved

## Governance

This issue is intended to collect the initial problem statement and benchmark plan.

Once Maka moves under Apache Incubator governance, the benchmark results and naming decision should be brought to the project’s `dev@` mailing list before implementation. The mailing-list thread should be treated as the authoritative project-level discussion, with links between that thread and this issue for continuity.

简体中文

## 问题

Maka 将跨平台的命令执行工具以 `Bash` 这一名称暴露给模型，但该名称并不能准确描述实际执行契约。

当前实现会通过平台选定的 shell 执行命令：

- POSIX 平台使用 `/bin/sh`。
- Windows 可能使用 PowerShell 7、Windows PowerShell 5.1 或 `cmd.exe`。
- 工具描述已经必须明确提醒 Windows 上的模型不要生成 Bash 语法。

相关证据：

- [`Bash` 是模型可见的工具名称](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/runtime/src/shell-tools.ts#L90-L100)
- [POSIX 命令通过 `/bin/sh` 执行](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/runtime/src/shell-detect.ts#L175-L176)
- [Windows 可能使用 PowerShell 或 `cmd.exe`](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/runtime/src/shell-detect.ts#L179-L193)

这带来了两个问题：

1. `Bash` 在 Windows 上具有明显误导性，即使在 POSIX 平台也不完全准确。
2. 工具名称可能影响模型的先验认知、工具选择、参数生成和失败恢复。对人类而言语义更准确的名称，不一定对所有模型都有更好的效果。

因此，本提案建议先对 `Bash`、`shell` 和 `exec` 进行 benchmark，再决定是否重命名。

## 需要做出的决策

开展可复现、受控的三臂消融实验：

```text
Bash — 当前基线
shell — 与平台无关的 shell 命令术语
exec — 简短的执行操作术语
```

Benchmark 应判断候选名称能否改善或至少保持 agent 表现，并且其收益是否足以承担兼容和迁移成本。

本 issue 不预设最终一定要改名。

## 候选名称的权衡

| 候选名称 | 优点 | 风险 |
| --- | --- | --- |
| `Bash` | 模型已有较强熟悉度；无需迁移 | 无法准确描述所有受支持的 shell，尤其会误导 Windows 用户和模型 |
| `shell` | 能准确表达跨平台抽象 | 可能与服务商原生 shell 能力及外部工具名称产生语义重叠 |
| `exec` | 简短，也是 agent 工具中常见的命名 | Maka 已将 [`exec` 用作 code mode 编排单元](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/runtime/src/ai-sdk-backend.ts#L1257-L1273)，因此需要明确解决名称冲突 |

## Benchmark 设计

### 受控变量

唯一应当变化的变量是模型可见的函数名称：

- 使用相同的 system prompt；
- 使用相同的工具描述；
- 使用相同的参数 schema；
- 使用相同的 runtime 和权限行为；
- 使用相同的模型和服务商配置；
- 使用相同的任务顺序和预算。

不能同时向模型暴露旧名称和候选名称。提供多个等价的命令工具会改变工具表面，使实验失去可比性。

### 评测套件

应同时采用针对性评测和端到端评测。

#### 1. 工具路由套件

应包含以下任务：

- 明确需要运行 shell 命令；
- 使用 `Read`、`Glob` 或 `Grep` 比运行 shell 更合适；
- 不需要调用任何工具；
- 命令失败后需要模型恢复；
- 覆盖前台、后台和 PTY 行为；
- prompt 中包含 “bash”“shell” 或 “execute”等词，但并不一定需要调用命令工具。

该套件应衡量名称是否改变模型的首次动作选择，或者导致不必要的命令执行。

#### 2. 跨平台命令套件

至少在以下环境运行等价任务：

- 使用 `/bin/sh` 的 POSIX 环境；
- 使用 Maka 所选 shell 的 Windows 环境。

需要同时衡量工具选择和 shell 方言正确性。重新审视 `Bash` 名称的主要动机正是跨平台准确性，因此 Windows 评测不可缺少。

#### 3. 端到端 Agent Benchmark

使用 Maka 现有评测基础设施开展配对的端到端 benchmark。

Terminal-Bench 可以维持与已有结果的连续性，但不应作为唯一评测套件：Maka 已有报告指出，Terminal-Bench 2.1 对前沿 agent harness 的区分空间有限。要隔离工具名称的影响，仍然需要针对性套件。

### 模型覆盖范围

应覆盖有代表性的受支持模型家族，而不是仅根据单一服务商得出结论。条件允许时，应固定准确的模型 ID 或快照。

每组配对实验必须保持模型和服务商一致。

### 指标

主要指标：

- 任务成功率；
- 针对性套件中的首次工具/动作选择正确率。

次要指标：

- 无效或格式错误的工具调用；
- 不必要的 shell 调用；
- shell 方言错误；
- 命令修复或重试次数；
- 工具调用和执行轮数；
- token、延迟和成本；
- 权限或 sandbox 流程回归。

### 实验纪律

- 在查看实验结果前预先登记主要指标和非劣效阈值。
- 对任务进行配对；服务商存在随机性时进行重复试验。
- 随机化或平衡不同实验臂的执行顺序。
- 保留原始 trace 和失败分类。
- 发布实验 manifest、聚合结果和逐任务结果。
- 区分模型错误、服务商失败、harness 失败和 benchmark 超时。

## 决策规则

只有候选名称同时满足以下条件时才进行重命名：

1. 端到端任务成功率满足非劣效要求；
2. 不造成安全、权限、sandbox 或恢复行为回归；
3. 在有代表性的模型上改善或保持针对性工具选择表现；
4. 存在可行的兼容方案。

如果两个候选名称都不能满足这些条件，则保留 `Bash`。

如果 `exec` 表现最好，仍必须先解决它与 code mode `exec` 工具的现有冲突。

## 兼容和迁移范围

这不是简单的字符串替换。`Bash` 当前同时承担以下身份：

- 模型可见工具目录中的名称；
- 权限和 sandbox escalation 记录中的判别值；
- 持久化 runtime event 和 transcript replay 中的工具身份；
- usage 聚合与搜索键；
- CLI 和 Desktop 的展示信息；
- shell-run overlay 和后台任务恢复逻辑；
- 内置 skill、prompt、测试和文档中的工具名称。

例如：

- [sandbox 权限记录要求 `toolName === 'Bash'`](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/core/src/interaction-record-schema.ts#L145-L155)
- [UI 通过精确工具名称识别 shell-run 更新](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/ui/src/materialize.ts#L573-L584)
- [`Bash` 属于规范化的产品工具目录](https://github.com/maka-agent/maka-agent/blob/db9407a0fa72a950fa9ec57353794c0faf35291b/packages/core/src/tool-catalog.ts#L78-L88)

如果最终批准改名，实现应当：

- 只向模型暴露新的规范名称；
- 继续读取历史 `Bash` event 和 transcript；
- 保持权限及 sandbox 语义不变；
- 保持新旧名称之间的 usage 和搜索连续性；
- 避免在代码库中散布大量双名称条件判断；
- 将内部 shell 工具识别迁移到语义 metadata 或单一规范化边界；
- 明确外部集成及导入会话的兼容范围。

Benchmark 得出结论后，应通过单独的 issue 或 PR 处理实现和迁移。

## 交付物

- [ ] 可复现的三臂 benchmark manifest
- [ ] 针对性工具路由与跨平台任务套件
- [ ] 有代表性的多模型评测结果
- [ ] 逐任务结果及失败分类
- [ ] 最终建议：保留 `Bash`、改为 `shell` 或改为 `exec`
- [ ] 推荐方案的兼容设计
- [ ] 批准改名后的后续实现 issue

## 治理

本 issue 用于收集初始问题陈述和 benchmark 计划。

Maka 转入 Apache Incubator 治理后，应在实现前将 benchmark 结果和命名决策提交到项目的 `dev@` 邮件列表讨论。邮件列表线程应作为项目级决策的权威讨论记录，并与本 issue 相互链接以保持上下文连续。

Drafted with Codex assistance and reviewed by M4n5ter, who takes responsibility for the proposal.

Start by reading packages/runtime/src/shell-tools.ts and packages/runtime/src/shell-detect.ts, then locate Maka’s existing evaluation infrastructure. Build the three-arm manifest and targeted cross-platform suites with identical tool behavior, preserve raw traces and failure classifications, and finish with reproducible results plus a keep-or-rename recommendation and compatibility design.

Written by the indexing model from the issue text.