jose-compu / jose-compu/funes.cpp
[LogosDB] Add memory consolidation for duplicate detection
- Dominant language
- C++
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Implement memory consolidation to detect and merge duplicate or similar memories.
## Problem
Teaching the same fact multiple times creates duplicate entries:
id=1: "My commute is 42 minutes"
id=2: "My commute takes 42 minutes"
id=3: "It takes 42 minutes to commute"
This wastes space and can confuse the compressor.
## Proposed Solution
### Option 1: Pre-insert duplicate detection
- Before teaching, search for similar embeddings
- If similarity > threshold, skip or update existing
### Option 2: Periodic consolidation
- Background task or manual /consolidate command
- Find clusters of similar embeddings
- Merge or keep only the best representative
### Option 3: Update-on-duplicate
- If similar memory exists, update it instead of creating new
- Preserves ID, updates text/timestamp/embedding
## Implementation Sketch
```cpp
// Pre-insert check
auto existing = logosdb_search(mem_db, embedding, dim, 1, &err);
if (existing && logosdb_result_count(existing) > 0) {
float score = logosdb_result_score(existing, 0);
if (score > 0.95) {
// Update instead of insert
uint64_t id = logosdb_result_id(existing, 0);
logosdb_update(mem_db, id, ...);
}
}
```
## CLI Additions
- /consolidate - Run manual deduplication
- --semantic-memory-dedup-threshold <0-1> - Similarity threshold
- --semantic-memory-auto-dedup - Enable automatic deduplication
## Acceptance Criteria
- [ ] Implement similarity check before insert
- [ ] Add /consolidate command for manual cleanup
- [ ] Add deduplication threshold option
- [ ] Add auto-dedup flag option
- [ ] Report deduplication stats in /stats
- [ ] Test duplicate detection accuracy
- [ ] Document in SPECS.md
## Dependencies
- Requires LogosDB v0.2.0 (issue #8)
- Benefits from memory ID tracking (issue #10)
## Related Files
- tools/cli/cli.cpp - /teach implementation
- SPECS.md - documentation
Contributor guide
Assessment
This issue has not been assessed yet.