MaartenGr / MaartenGr/BERTopic
merge_models create new outlier topic
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi Maarten,
I've been reducing outliers by running another topic model exclusively on them, and it has been effective. However, when attempting to merge the models, a new topic (-1) is created that doesn't exist in all models. For instance, in my initial run, I obtained 83 documents belonging to topic -1. So, I ran it again only on the topic -1 documents, resulting in the addition of two topics that covered all outlier documents, effectively eliminating outliers. However, upon merging the models using 'merge_models', I noticed a new model with topic -1 doubled in size compared to the first run, with 166 documents now belonging to topic -1. Do you have any insights into what might have caused this discrepancy?
first model:

second model on the outliers alone:

and the merge_model:

thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported workflow: fit a first topic model, fit a second model on its topic -1 documents, then run merge_models. Compare the topic assignments and outlier counts before and after merging. Done means identifying why topic -1 is duplicated or recreated and documenting or correcting the resulting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100