activeloopai / activeloopai/hivemind

embeddings: alternation patterns (a|b|c) skip the embedding daemon and fall back to lexical-only

Đang mở
#86 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
TypeScript
Star
1.6k
Fork
107
Merge trung bình
17 giờ 30 phút
Pull request đã merge (30 ngày)
6

Mô tả

## Severity
Medium-high — limits the embedding feature's main UX (synonym recall is exactly the case embeddings are designed for).

## Where
- `src/hooks/grep-direct.ts:36-50` — `patternIsSemanticFriendly`
- `src/shell/grep-interceptor.ts:43-50` — same function (duplicated)

## Repro
On any workspace with embedding rows, ask Claude Code:

```
HIVEMIND_DEBUG=1 claude -p 'Search hivemind for sessions about silent data loss or concurrent-writer corruption. Use grep -r over the memory path.'
```

CC issues a Bash `grep -r` with an alternation pattern like `silent data loss|data loss|concurrent writer|writer corruption|race condition|lost write`. The debug log (`~/.deeplake/hook-debug.log`) shows **zero** `summary_embedding <#> ARRAY[…]` queries — only lexical `ILIKE` matches and path browses. CC's answer is correct only when the literal keywords match a summary; it misses paraphrases.

## Root cause
`patternIsSemanticFriendly` rejects any pattern with more than 1 regex metacharacter, and `|` counts as a metacharacter:

```ts
const meta = pattern.match(/[|()\[\]{}+?^$\\]/g);
if (!meta) return true;
return meta.length <= 1;
```

When false, the dispatcher skips the embedding daemon call and `searchDeeplakeTables` runs the lexical-only branch.

But `|` is exactly the synonym-list use case where embeddings shine — different surface forms of the same concept. The original intent of the filter ("don't waste an embed call on regex like `[a-z]+\.com`") shouldn't reject synonym alternations.

## Proposed fix
Drop `|` from the metacharacter blacklist + cap the alternative count to guard against pathological inputs:

```ts
function patternIsSemanticFriendly(pattern: string, fixedString: boolean): boolean {
if (!pattern || pattern.length < 2) return false;
if (fixedString) return true;
const meta = pattern.match(/[()\[\]{}+?^$\\]/g); // dropped `|`
if (meta && meta.length > 1) return false;
// Still gate on alternative count to avoid 50-element synonym blasts:
const altCount = pattern.split("|").length;
return altCount <= 8;
}
```

Apply to both `grep-direct.ts` and `grep-interceptor.ts` (or extract into a shared helper while we're at it).

## Acceptance
After fix, the same repro prompt should produce ≥ 1 `summary_embedding <#> ARRAY[…]` query in the debug log, and CC's answer should include sessions matched by paraphrase (not just literal keyword).

## Out of scope
- Recommending Grep tool over Bash grep in the SessionStart prompt (Bash grep already routes through `handleGrepDirect`, which is the embedding path).
- Removing the duplication between the two `patternIsSemanticFriendly` definitions (separate refactor).

Found during the embedding_generation branch validation, see commits 4e956fa + f9f10f3.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.