MaartenGr / MaartenGr/BERTopic
approximate_distribution returns only 0s
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi Maarten,
Sorry for keeping bugging you! I really enjoy utilizing BERTopic for my project!
A new question just occurred to me. I need to know topic distribution of each document so I follow your instruction to implement approximate_distribution, but it returns with a ndarray containing nothing but 0s.
get_document_info does show the dominant topic and its probability in each document.
Following is the code I use.
```
umap_model = UMAP(n_neighbors=100, n_components=10, min_dist=0.1, metric='cosine', random_state=42)
hdbscan_model = HDBSCAN(min_cluster_size=20, metric='euclidean', cluster_selection_method='eom', prediction_data=True,
min_samples=5)
from sklearn.feature_extraction.text import CountVectorizer
vectorizer_model = CountVectorizer(ngram_range=(1, 2), stop_words=stopWord)
from bertopic.vectorizers import ClassTfidfTransformer
ctfidf_model = ClassTfidfTransformer(seed_words=['台独','蔡英文', '佩洛西'], bm25_weighting=True, reduce_frequent_words=True)
topic_model = BERTopic(embedding_model="distiluse-base-multilingual-cased-v1",
language="multilingual",
verbose=True,
calculate_probabilities=True,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
vectorizer_model=vectorizer_model,
# representation_model=representation_model,
ctfidf_model=ctfidf_model,
nr_topics=22,
n_gram_range=(1, 2),
top_n_words=15,
min_topic_size=200)
topics, probs = topic_model.fit_transform(sampled_data2_as_strings, embeddings)
topic_distr, _ = topic_model.approximate_distribution(sampled_data2_as_strings)
print(topic_distr)
```
[[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
...
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]
[0. 0. 0. ... 0. 0. 0.]]
The result just doesn't make sense...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported behavior with the provided BERTopic configuration, then inspect approximate_distribution alongside fit_transform and the returned probabilities. Compare the output for the supplied documents and determine why every distribution entry is zero. Done means the behavior is corrected or its cause is clearly documented, with a regression test for the reproduced case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100