MaartenGr / MaartenGr/BERTopic
MMR doesn't work/ doesn't make a change
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi Maarten,
Thank you once again for this amazing package. I used it for my master's thesis and several projects for my job at the university, and it's a lifesaver compared to other topic modeling techniques I tried.
That being said, I have run the model on about 200k tweets and many topics have quite a lot of repeating words. I have used the following code to add a representation model with different parameters (starting from .3 to .9) but the results are still the same.
```
mmr = MaximalMarginalRelevance(diversity=.9)
representation_model = {
"mmr": mmr
}
topic_model = BERTopic (
umap_model=UMAP(),
hdbscan_model=hdbscan_model,
vectorizer_model=vectorizer_model,
embedding_model=embedding_model,
top_n_words=10,
language='english',
verbose=True,
representation_model=representation_model
)
```
Here are a couple of examples:
> Representation:
> australia,australias,australian,auspol,renewable,nsw,energy,government,queensland,coal
> MMR:
> australia,australias,australian,auspol,renewable,nsw,energy,government,queensland,coal
Not only the results are the same, but plurals such as australia and australians are not merged. Could you please guide me on how to move further?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported result with the shown BERTopic configuration and the MaximalMarginalRelevance representation model, varying diversity as described. Trace how representation_model is applied to topic words and determine why the MMR output remains unchanged; done means explaining or correcting that behavior and verifying the repeated-word example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100