MaartenGr / MaartenGr/BERTopic

hierarchical_topics() produce incorrect output when three topics have the same distance

Open
#1,907 4 comments 0 reactions 1 assignee View on GitHub

@azikoss is already working on this.

Since Apr 10, 2024.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi there,

I have noticed that hierarchical_topics(...) method produces incorrect results when three or more topics have the same (tf-idf) distances. Let me illustrative it with an example.

from umap import UMAP
from bertopic import BERTopic

docs = (
    ["banana"] * 300 
    + ["banana apple"] * 300 
    + ["pear"] * 300 
    + ["lemon"] * 300 
    + ["clock"] * 300 
)

model = BERTopic(umap_model=UMAP(random_state=42))
topics, probs = model.fit_transform(docs)
hr = model.hierarchical_topics(docs)
hr

This outputs
image

The cluster with Parent_ID == 8 includes topics [1, 2, 3] but topic 3 is not mentioned in either the left or right child or their childs.

Why is it happening?
The flat structure is created in each iteration. There is no guarantee that the new cluster will contain only two topics while the code that follows presumes that.

What should be the expected behavior?
I think that a new cluster should emerge. Essentially, the structure should look like this

Parent_Id, Child_Left_ID, Child_Right_ID
     8               1               11
    11               2               3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.