MaartenGr / MaartenGr/BERTopic

approximate_distribution returns only 0s

Open
#1,995 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi Maarten,

Sorry for keeping bugging you! I really enjoy utilizing BERTopic for my project!

A new question just occurred to me. I need to know topic distribution of each document so I follow your instruction to implement approximate_distribution, but it returns with a ndarray containing nothing but 0s.

get_document_info does show the dominant topic and its probability in each document.

Following is the code I use.

```
umap_model = UMAP(n_neighbors=100, n_components=10, min_dist=0.1, metric='cosine', random_state=42)
hdbscan_model = HDBSCAN(min_cluster_size=20, metric='euclidean', cluster_selection_method='eom', prediction_data=True,
min_samples=5)
from sklearn.feature_extraction.text import CountVectorizer
vectorizer_model = CountVectorizer(ngram_range=(1, 2), stop_words=stopWord)
from bertopic.vectorizers import ClassTfidfTransformer
ctfidf_model = ClassTfidfTransformer(seed_words=['台独','蔡英文', '佩洛西'], bm25_weighting=True, reduce_frequent_words=True)

topic_model = BERTopic(embedding_model="distiluse-base-multilingual-cased-v1",
language="multilingual",
verbose=True,
calculate_probabilities=True,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
vectorizer_model=vectorizer_model,
# representation_model=representation_model,
ctfidf_model=ctfidf_model,
nr_topics=22,
n_gram_range=(1, 2),
top_n_words=15,
min_topic_size=200)
topics, probs = topic_model.fit_transform(sampled_data2_as_strings, embeddings)

topic_distr, _ = topic_model.approximate_distribution(sampled_data2_as_strings)
print(topic_distr)
```

[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]

The result just doesn't make sense...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported behavior with the provided BERTopic configuration, then inspect approximate_distribution alongside fit_transform and the returned probabilities. Compare the output for the supplied documents and determine why every distribution entry is zero. Done means the behavior is corrected or its cause is clearly documented, with a regression test for the reproduced case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.