explosion / explosion/spaCy

Filter duplicate vectors when pruning vectors

Open
#5,397 2 comments 2 reactions 0 assignees View on GitHub
bug feat / vectors
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour

When prioritizing vectors to keep, `Vocab.prune_vectors` doesn't handle existing duplicates from `key2row` well. By sorting/prioritizing by values from `key2row`, which may contain duplicate values, `prune_vectors` may keep multiple copies of the same vector.

Fix:

* remove duplicate vectors from `indices` and adjust `keys` accordingly
* preserve relevant duplicate `key2row` keys (which is not compatible with the keys truncation from #5348, which is overly simple; I think you have to re-add the duplicate rows to vectors after initialization with `Vectors.add(row=)`)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.