jose-compu / jose-compu/funes.cpp

[LogosDB] Add memory consolidation for duplicate detection

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
C++
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Implement memory consolidation to detect and merge duplicate or similar memories.

## Problem
Teaching the same fact multiple times creates duplicate entries:
id=1: "My commute is 42 minutes"
id=2: "My commute takes 42 minutes"
id=3: "It takes 42 minutes to commute"

This wastes space and can confuse the compressor.

## Proposed Solution

### Option 1: Pre-insert duplicate detection
- Before teaching, search for similar embeddings
- If similarity > threshold, skip or update existing

### Option 2: Periodic consolidation
- Background task or manual /consolidate command
- Find clusters of similar embeddings
- Merge or keep only the best representative

### Option 3: Update-on-duplicate
- If similar memory exists, update it instead of creating new
- Preserves ID, updates text/timestamp/embedding

## Implementation Sketch
```cpp
// Pre-insert check
auto existing = logosdb_search(mem_db, embedding, dim, 1, &err);
if (existing && logosdb_result_count(existing) > 0) {
float score = logosdb_result_score(existing, 0);
if (score > 0.95) {
// Update instead of insert
uint64_t id = logosdb_result_id(existing, 0);
logosdb_update(mem_db, id, ...);
}
}
```

## CLI Additions
- /consolidate - Run manual deduplication
- --semantic-memory-dedup-threshold <0-1> - Similarity threshold
- --semantic-memory-auto-dedup - Enable automatic deduplication

## Acceptance Criteria
- [ ] Implement similarity check before insert
- [ ] Add /consolidate command for manual cleanup
- [ ] Add deduplication threshold option
- [ ] Add auto-dedup flag option
- [ ] Report deduplication stats in /stats
- [ ] Test duplicate detection accuracy
- [ ] Document in SPECS.md

## Dependencies
- Requires LogosDB v0.2.0 (issue #8)
- Benefits from memory ID tracking (issue #10)

## Related Files
- tools/cli/cli.cpp - /teach implementation
- SPECS.md - documentation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.