Filter duplicate vectors when pruning vectors
Open
bug
feat / vectors
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## How to reproduce the behaviour
When prioritizing vectors to keep, `Vocab.prune_vectors` doesn't handle existing duplicates from `key2row` well. By sorting/prioritizing by values from `key2row`, which may contain duplicate values, `prune_vectors` may keep multiple copies of the same vector.
Fix:
* remove duplicate vectors from `indices` and adjust `keys` accordingly
* preserve relevant duplicate `key2row` keys (which is not compatible with the keys truncation from #5348, which is overly simple; I think you have to re-add the duplicate rows to vectors after initialization with `Vectors.add(row=)`)
Contributor guide
Assessment
This issue has not been assessed yet.