MaartenGr / MaartenGr/BERTopic

BERTopic - Topic reduction produced 18 topics when nr_topics=96

Open
#2,099 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Have you searched existing issues? 🔎

- [X] I have searched and found no existing issues

### Desribe the bug

I am stuck with the bug: set nr_topics=96 (min_cluster_size=60, min_samples=30) then have "BERTopic - Topic reduction - Reduced number of topics from 18 to 18".
Should it be like "BERTopic - Topic reduction - Reduced number of topics from 96 to 18" or something?
Do you have any ideas on how to get rid of this "Topic reduction" step?

### Reproduction

## Train model
```python
representation_model = {
"Main": KeyBERTInspired(),
"POS": [
PartOfSpeech("en_core_web_sm", pos_patterns=pos_patterns),
MaximalMarginalRelevance(diversity=.4)
],
}

# Vectorizer runs after embeddings are generated, it's only affect topic's words representation, therefore we use stop_words and lemmatizer here
vectorizer_model = CountVectorizer(
min_df=MIN_DF,
max_df=MAX_DF,
ngram_range=NGRAM_RANGE,
stop_words='english',
tokenizer=LemmaTokenizer(),
)

umap_model = UMAP(
n_neighbors=15,
n_components=50,
min_dist=0.0,
metric='cosine',
random_state=42 # UMAP is stochastic algorithm, so don't forget to set seed to make results reproduciable
)

hdbscan_model = HDBSCAN(
min_cluster_size=60, min_samples=30, # Use values from TMT
metric='euclidean',
cluster_selection_method='eom',
prediction_data=True
)

topic_model = BERTopic(
nr_topics=96, # Use value from TMT
vectorizer_model=vectorizer_model,
representation_model=representation_model,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
verbose=True
)

topics, ini_probs = topic_model.fit_transform(splited_data)
```
```log
2024-07-26 17:20:18,830 - BERTopic - Embedding - Transforming documents to embeddings.
Batches: 100%
 1563/1563 [00:13<00:00, 141.83it/s]
2024-07-26 17:20:33,447 - BERTopic - Embedding - Completed ✓
2024-07-26 17:20:33,448 - BERTopic - Dimensionality - Fitting the dimensionality reduction algorithm
2024-07-26 17:20:35,428 - BERTopic - Dimensionality - Completed ✓
2024-07-26 17:20:35,441 - BERTopic - Cluster - Start clustering the reduced embeddings
2024-07-26 17:20:37,022 - BERTopic - Cluster - Completed ✓
2024-07-26 17:20:37,023 - BERTopic - Representation - Extracting topics from clusters using representation models.
2024-07-26 17:21:25,257 - BERTopic - Representation - Completed ✓
2024-07-26 17:21:25,258 - BERTopic - Topic reduction - Reducing number of topics
2024-07-26 17:21:25,261 - BERTopic - Topic reduction - Reduced number of topics from 18 to 18
```

### BERTopic Version

0.16.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported BERTopic 0.16.3 behavior with the supplied BERTopic, UMAP, HDBSCAN, and nr_topics=96 configuration, then inspect the topic-reduction stage reached by fit_transform. Determine whether the log reflects expected reduction semantics or an incorrect count, and document or test the observed behavior once the expected result is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.