feat: semantic codebase_search tool (BM25 or embedding index)
- Dominant language
- Rust
- Stars
- 41k
- Forks
- 3.6k
- Avg merge
- 13h 59m
- Merged PRs (30d)
- 299
Description
## Problem
CodeWhale search today is regex plus fuzzy filename lookup (`grep_files`, `file_search`, `project_map`). There is no FTS5, BM25, or embedding-backed index (`docs/TOOL_LIFECYCLE.md` notes this explicitly). Config exposes `prefer_codebase_search`, but the `codebase_search` tool is not registered in the model-visible catalog, so agents burn turns on grep walks and miss concepts that are not string-matchable — the largest discovery gap vs Cursor `@Codebase` / semantic search.
## Acceptance criteria
- Agent can call `codebase_search(query)` and receive ranked file/chunk hits with bounded excerpts.
- `prefer_codebase_search` (or equivalent harness posture) enables tool registration when an index backend is available.
- Index build/update is documented (scope, exclusions, refresh triggers) and respects future privacy controls.
- Tool behavior is documented in `docs/TOOL_SURFACE.md` (or successor) alongside existing search-family tools.
- Tests cover empty index, hit ranking, byte caps, and graceful fallback when indexing is disabled.
Contributor guide
Research direction
Start with docs/TOOL_LIFECYCLE.md and trace the existing search-family tools, the model-visible catalog, and the prefer_codebase_search configuration. Define the index scope, exclusions, refresh triggers, bounded excerpts, ranking, and disabled-index fallback before implementing. Update docs/TOOL_SURFACE.md and add coverage for empty indexes, ranking, byte caps, and fallback behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100