改进语音模式对专有名词和技术词的识别准确度
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 38/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- typescript
- Domain
- audio-video-rtc, cli
Research direction
Start by reading packages/realtime/src/backend/stepfun-stateless.ts and src/runtime/build-voice-runtime.ts, then run the existing step voice flow and inspect its configuration and CLI entry points. Determine where vocabulary can be loaded and passed through the voice pipeline. Done means users can configure and manage a vocabulary, step voice loads it, and the documented acceptance examples are handled with clear limitations.
Written by the indexing model from the issue text.
Description
背景
当前 step voice 已经可以完成实时语音编程,但在编程场景里,用户经常会说到大量专有名词和精确术语,例如:
- GitHub / GitHub Actions
- MCP / OpenTUI / VAD / AEC
- npm 包名、仓库名、文件名、变量名、命令名
- 公司内部项目名、服务名、缩写
这些词一旦被 ASR 识别错,后续 coding agent 可能仍能靠上下文猜到大意,但涉及精确路径、命令、符号或仓库名时,误识别会明显影响可用性。
当前观察
从当前实现看,语音链路主要是把音频通过 realtime backend 发送,并在 response.create 中传入 instructions、history、tools、voice、speed_ratio 等信息:
packages/realtime/src/backend/stepfun-stateless.ts中发送input_audio_buffer.append/input_audio_buffer.commit- 同文件的
requestResponse组装response.create,包含instructions、history、tools等 src/runtime/build-voice-runtime.ts中有整体voiceInstructions,但没有看到独立的 ASR 热词、专有词词表、自定义 vocabulary 配置
也就是说,目前更像是依赖通用实时语音模型的识别能力和上下文推断,而不是显式告诉 ASR:“这些词在本项目里很重要”。
参考体验
本地查看“闪电说”时,能看到它在“技能 > 语音输入”里提供了“加载用户词典”能力,说明文案是:
使用你定义的个人词库,提升语音识别的术语准确度
这类能力对编程语音输入很有参考价值。Step Realtime CLI 不一定需要完整复制一个复杂的“技能/记忆”系统,但建议至少提供一个轻量的用户词典或项目词库入口。
建议方案
可以考虑分阶段实现:
- 增加一个轻量配置项,例如:
{
"voice": {
"recognition": {
"vocabulary": [
"GitHub",
"GitHub Actions",
"OpenTUI",
"MCP",
"stepfun-ai/Step-Realtime-CLI"
]
}
}
}
-
如果 StepFun realtime backend 支持 ASR 热词 / phrase hints / custom vocabulary,优先将这些词下发到 backend 的识别参数中。
-
如果 backend 暂时没有专门字段,可以先做降级方案:
- 把词库渲染进 voice instructions,提示模型在听到相近发音时优先映射为这些术语;
- 在 transcript 进入 coding agent 前做一层轻量术语纠错/归一化;
- 允许项目级词库从仓库配置或
AGENTS.md/ README / package metadata 中自动提取候选词。
-
提供 CLI 管理入口,例如:
step voice vocab add GitHub
step voice vocab add "GitHub Actions"
step voice vocab list
step voice vocab remove GitHub
验收标准
- 用户可以配置一组语音识别专有词或项目词库。
step voice能在启动时加载这些词。- 像 “GitHub”、“GitHub Actions”、“OpenTUI”、“MCP” 这类技术词,在普通语音输入中更稳定地保留正确写法。
- 文档说明该能力的边界:词库用于提升术语识别和归一化,不保证所有精确拼写都 100% 正确。
价值
这会让语音编程从“能听懂自然语言意图”,进一步提升到“能稳定处理编程场景中的精确术语”。对语音修改代码、读文件、执行命令、描述 GitHub 工作流等场景都很关键。
- Dominant language
- TypeScript
- Stars
- 54
- Forks
- 20
- Avg merge
- 2h 58m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from stepfun-ai/Step-Code
-
status/needs-triage type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
stepfun-ai/Step-Code#85 · 1 comment ·
-
status/needs-triage type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
stepfun-ai/Step-Code#84 ·
-
status/needs-triage type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
stepfun-ai/Step-Code#83 ·
-
status/needs-triage type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
stepfun-ai/Step-Code#81 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
stepfun-ai/Step-Code#80 ·
All issues in stepfun-ai/Step-Code
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
area:tools bug good first issue help wanted priority:P2
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
TaewoooPark/Motifcode#14 ·
-
bug 🐞
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
[Bounty proposal] fix(web): memory insights count an evening memory on the next day ($25 proposed) Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
BasedHardware/omi#15320 ·
-
Difficulty 2/5 Half a day Newbie friendliness 78/100
vercel/vercel-plugin#199 ·