Hmbown / Hmbown/Codewhale

feat: semantic codebase_search tool (BM25 or embedding index)

Open
#4,066 0 comments 0 reactions 0 assignees View on GitHub
documentation enhancement tools
Dominant language
Rust
Stars
41k
Forks
3.6k
Avg merge
13h 59m
Merged PRs (30d)
299

Description

## Problem

CodeWhale search today is regex plus fuzzy filename lookup (`grep_files`, `file_search`, `project_map`). There is no FTS5, BM25, or embedding-backed index (`docs/TOOL_LIFECYCLE.md` notes this explicitly). Config exposes `prefer_codebase_search`, but the `codebase_search` tool is not registered in the model-visible catalog, so agents burn turns on grep walks and miss concepts that are not string-matchable — the largest discovery gap vs Cursor `@Codebase` / semantic search.

## Acceptance criteria

- Agent can call `codebase_search(query)` and receive ranked file/chunk hits with bounded excerpts.
- `prefer_codebase_search` (or equivalent harness posture) enables tool registration when an index backend is available.
- Index build/update is documented (scope, exclusions, refresh triggers) and respects future privacy controls.
- Tool behavior is documented in `docs/TOOL_SURFACE.md` (or successor) alongside existing search-family tools.
- Tests cover empty index, hit ranking, byte caps, and graceful fallback when indexing is disabled.

Contributor guide

Open the contributing guide

Research direction

Start with docs/TOOL_LIFECYCLE.md and trace the existing search-family tools, the model-visible catalog, and the prefer_codebase_search configuration. Define the index scope, exclusions, refresh triggers, bounded excerpts, ranking, and disabled-index fallback before implementing. Update docs/TOOL_SURFACE.md and add coverage for empty indexes, ranking, byte caps, and fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.