MaartenGr / MaartenGr/BERTopic
hierarchical_topics() produce incorrect output when three topics have the same distance
@azikoss is already working on this.
Since Apr 10, 2024.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi there,
I have noticed that hierarchical_topics(...) method produces incorrect results when three or more topics have the same (tf-idf) distances. Let me illustrative it with an example.
from umap import UMAP
from bertopic import BERTopic
docs = (
["banana"] * 300
+ ["banana apple"] * 300
+ ["pear"] * 300
+ ["lemon"] * 300
+ ["clock"] * 300
)
model = BERTopic(umap_model=UMAP(random_state=42))
topics, probs = model.fit_transform(docs)
hr = model.hierarchical_topics(docs)
hr
This outputs
The cluster with Parent_ID == 8 includes topics [1, 2, 3] but topic 3 is not mentioned in either the left or right child or their childs.
Why is it happening?
The flat structure is created in each iteration. There is no guarantee that the new cluster will contain only two topics while the code that follows presumes that.
What should be the expected behavior?
I think that a new cluster should emerge. Essentially, the structure should look like this
Parent_Id, Child_Left_ID, Child_Right_ID
8 1 11
11 2 3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.