MaartenGr / MaartenGr/PolyFuzz
TFIDF min_similarity not applied
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 803
- Forks
- 72
- PR merge metrics
- No merged PRs in 30d
Description
When using the TFIDF model the min_similiary parameter seems not to be applied to the results.
Minimal Example that reproduces the problem (polyfuzz 0.4.0):
from polyfuzz import PolyFuzz
from polyfuzz.models import TFIDF
if __name__ == "__main__":
token_list = [
"Stoltenbergs",
"Ansage",
"Putin",
"Nato",
"Drohungen",
"Russlands",
"Nato",
"Unterstützung",
"Ukraine",
"Stoltenberg",
"Putin",
"Nato",
]
matcher = TFIDF(n_gram_range=(3, 3), min_similarity=0.9)
model = PolyFuzz(matcher)
model.match(token_list)
model.group()
matches = model.get_matches()
print(matches)
When running the code the following output is generated, but the rows 4 and 7 should have a Similarity score of 0, if I understand the documentation correctly.
The minimum similarity between strings, otherwise return 0 similarity
I would expect the rows with a Similarity of < 0.9 to have a Similarity of 0 and a To value of None.
Output:
From To Similarity Group
0 Stoltenbergs Stoltenberg 0.932 Stoltenbergs
1 Ansage None 0.000 None
2 Putin Putin 1.000 Putin
3 Nato Nato 1.000 Nato
4 Drohungen Unterstützung 0.091 Unterstützung
5 Russlands None 0.000 None
6 Nato Nato 1.000 Nato
7 Unterstützung Drohungen 0.091 Drohungen
8 Ukraine None 0.000 None
9 Stoltenberg Stoltenbergs 0.932 Stoltenbergs
10 Putin Putin 1.000 Putin
11 Nato Nato 1.000 Nato
In case I'm using the library wrong, how would I be able to get only results with a similarity higher than 0.9?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the TFIDF model's handling of the min_similarity parameter and reproduce the issue using the minimal PolyFuzz Python example. Done means matches below 0.9 have similarity 0 and a To value of None, while higher-scoring matches remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100