MaartenGr / MaartenGr/PolyFuzz

Grouping issue when TFIDF.min_similarity < link_min_similarity

Open
#40 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
803
Forks
72
PR merge metrics
No merged PRs in 30d

Description

In the code below (with output in attached picture) I perform a simple TFIDF matching of `["apple", "apples", "appl", "recal", "happy"]`.

The initial `min_similarity` is set to 0.2. The similarity of `happy` and `appl` is 0.24.

When grouping with a `link_min_similarity` of 0.5, `happy` should not belong in the `apples` group, though that's what happens in the output of `.get_matches()`, it is in the `apples` group.

It appears it is not in the cluster though.

![grouping](https://user-images.githubusercontent.com/17087829/180418077-59157cb4-6c5e-42e9-a2e6-1ce1389b90a2.png)

Plain text code:

```python
from polyfuzz import PolyFuzz
from polyfuzz.models import TFIDF

from_list = ["apple", "apples", "appl", "recal", "happy"]
matcher = TFIDF(min_similarity=0.2)
model = PolyFuzz(matcher).match(from_list)
cm = model.cluster_mappings
model.group(link_min_similarity=0.5, group_all_strings=True)
print(model.get_matches())
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied Python reproduction using PolyFuzz, TFIDF, group(link_min_similarity=0.5, group_all_strings=True), and get_matches(). Inspect how grouping handles the happy/appl similarity versus the link threshold. Done means happy is not reported in the apples group when its similarity is below link_min_similarity.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.