activeloopai / activeloopai/hivemind

embeddings: alternation patterns (a|b|c) skip the embedding daemon and fall back to lexical-only

Aberta
#86 0 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
TypeScript
Estrelas
1.6k
Forks
107
Merge médio
17h 30min
PRs com merge (30d)
6

Descrição

## Severity
Medium-high — limits the embedding feature's main UX (synonym recall is exactly the case embeddings are designed for).

## Where
- `src/hooks/grep-direct.ts:36-50` — `patternIsSemanticFriendly`
- `src/shell/grep-interceptor.ts:43-50` — same function (duplicated)

## Repro
On any workspace with embedding rows, ask Claude Code:

```
HIVEMIND_DEBUG=1 claude -p 'Search hivemind for sessions about silent data loss or concurrent-writer corruption. Use grep -r over the memory path.'
```

CC issues a Bash `grep -r` with an alternation pattern like `silent data loss|data loss|concurrent writer|writer corruption|race condition|lost write`. The debug log (`~/.deeplake/hook-debug.log`) shows **zero** `summary_embedding <#> ARRAY[…]` queries — only lexical `ILIKE` matches and path browses. CC's answer is correct only when the literal keywords match a summary; it misses paraphrases.

## Root cause
`patternIsSemanticFriendly` rejects any pattern with more than 1 regex metacharacter, and `|` counts as a metacharacter:

```ts
const meta = pattern.match(/[|()\[\]{}+?^$\\]/g);
if (!meta) return true;
return meta.length <= 1;
```

When false, the dispatcher skips the embedding daemon call and `searchDeeplakeTables` runs the lexical-only branch.

But `|` is exactly the synonym-list use case where embeddings shine — different surface forms of the same concept. The original intent of the filter ("don't waste an embed call on regex like `[a-z]+\.com`") shouldn't reject synonym alternations.

## Proposed fix
Drop `|` from the metacharacter blacklist + cap the alternative count to guard against pathological inputs:

```ts
function patternIsSemanticFriendly(pattern: string, fixedString: boolean): boolean {
if (!pattern || pattern.length < 2) return false;
if (fixedString) return true;
const meta = pattern.match(/[()\[\]{}+?^$\\]/g); // dropped `|`
if (meta && meta.length > 1) return false;
// Still gate on alternative count to avoid 50-element synonym blasts:
const altCount = pattern.split("|").length;
return altCount <= 8;
}
```

Apply to both `grep-direct.ts` and `grep-interceptor.ts` (or extract into a shared helper while we're at it).

## Acceptance
After fix, the same repro prompt should produce ≥ 1 `summary_embedding <#> ARRAY[…]` query in the debug log, and CC's answer should include sessions matched by paraphrase (not just literal keyword).

## Out of scope
- Recommending Grep tool over Bash grep in the SessionStart prompt (Bash grep already routes through `handleGrepDirect`, which is the embedding path).
- Removing the duplication between the two `patternIsSemanticFriendly` definitions (separate refactor).

Found during the embedding_generation branch validation, see commits 4e956fa + f9f10f3.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.