MaartenGr / MaartenGr/PolyFuzz

TFIDF min_similarity not applied

Open
#49 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
803
Forks
72
PR merge metrics
No merged PRs in 30d

Description

When using the TFIDF model the min_similiary parameter seems not to be applied to the results.

Minimal Example that reproduces the problem (polyfuzz 0.4.0):

from polyfuzz import PolyFuzz
from polyfuzz.models import TFIDF

if __name__ == "__main__":
    token_list = [
        "Stoltenbergs",
        "Ansage",
        "Putin",
        "Nato",
        "Drohungen",
        "Russlands",
        "Nato",
        "Unterstützung",
        "Ukraine",
        "Stoltenberg",
        "Putin",
        "Nato",
    ]

    matcher = TFIDF(n_gram_range=(3, 3), min_similarity=0.9)
    model = PolyFuzz(matcher)
    model.match(token_list)
    model.group()
    matches = model.get_matches()
    print(matches)

When running the code the following output is generated, but the rows 4 and 7 should have a Similarity score of 0, if I understand the documentation correctly.

The minimum similarity between strings, otherwise return 0 similarity

I would expect the rows with a Similarity of < 0.9 to have a Similarity of 0 and a To value of None.

Output:

             From             To  Similarity          Group
0    Stoltenbergs    Stoltenberg       0.932   Stoltenbergs
1          Ansage           None       0.000           None
2           Putin          Putin       1.000          Putin
3            Nato           Nato       1.000           Nato
4       Drohungen  Unterstützung       0.091  Unterstützung
5       Russlands           None       0.000           None
6            Nato           Nato       1.000           Nato
7   Unterstützung      Drohungen       0.091      Drohungen
8         Ukraine           None       0.000           None
9     Stoltenberg   Stoltenbergs       0.932   Stoltenbergs
10          Putin          Putin       1.000          Putin
11           Nato           Nato       1.000           Nato

In case I'm using the library wrong, how would I be able to get only results with a similarity higher than 0.9?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TFIDF model's handling of the min_similarity parameter and reproduce the issue using the minimal PolyFuzz Python example. Done means matches below 0.9 have similarity 0 and a To value of None, while higher-scoring matches remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.