MaartenGr / MaartenGr/BERTopic
BERTopic - Topic reduction produced 18 topics when nr_topics=96
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Have you searched existing issues? 🔎
- [X] I have searched and found no existing issues
### Desribe the bug
I am stuck with the bug: set nr_topics=96 (min_cluster_size=60, min_samples=30) then have "BERTopic - Topic reduction - Reduced number of topics from 18 to 18".
Should it be like "BERTopic - Topic reduction - Reduced number of topics from 96 to 18" or something?
Do you have any ideas on how to get rid of this "Topic reduction" step?
### Reproduction
## Train model
```python
representation_model = {
"Main": KeyBERTInspired(),
"POS": [
PartOfSpeech("en_core_web_sm", pos_patterns=pos_patterns),
MaximalMarginalRelevance(diversity=.4)
],
}
# Vectorizer runs after embeddings are generated, it's only affect topic's words representation, therefore we use stop_words and lemmatizer here
vectorizer_model = CountVectorizer(
min_df=MIN_DF,
max_df=MAX_DF,
ngram_range=NGRAM_RANGE,
stop_words='english',
tokenizer=LemmaTokenizer(),
)
umap_model = UMAP(
n_neighbors=15,
n_components=50,
min_dist=0.0,
metric='cosine',
random_state=42 # UMAP is stochastic algorithm, so don't forget to set seed to make results reproduciable
)
hdbscan_model = HDBSCAN(
min_cluster_size=60, min_samples=30, # Use values from TMT
metric='euclidean',
cluster_selection_method='eom',
prediction_data=True
)
topic_model = BERTopic(
nr_topics=96, # Use value from TMT
vectorizer_model=vectorizer_model,
representation_model=representation_model,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
verbose=True
)
topics, ini_probs = topic_model.fit_transform(splited_data)
```
```log
2024-07-26 17:20:18,830 - BERTopic - Embedding - Transforming documents to embeddings.
Batches: 100%
1563/1563 [00:13<00:00, 141.83it/s]
2024-07-26 17:20:33,447 - BERTopic - Embedding - Completed ✓
2024-07-26 17:20:33,448 - BERTopic - Dimensionality - Fitting the dimensionality reduction algorithm
2024-07-26 17:20:35,428 - BERTopic - Dimensionality - Completed ✓
2024-07-26 17:20:35,441 - BERTopic - Cluster - Start clustering the reduced embeddings
2024-07-26 17:20:37,022 - BERTopic - Cluster - Completed ✓
2024-07-26 17:20:37,023 - BERTopic - Representation - Extracting topics from clusters using representation models.
2024-07-26 17:21:25,257 - BERTopic - Representation - Completed ✓
2024-07-26 17:21:25,258 - BERTopic - Topic reduction - Reducing number of topics
2024-07-26 17:21:25,261 - BERTopic - Topic reduction - Reduced number of topics from 18 to 18
```
### BERTopic Version
0.16.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported BERTopic 0.16.3 behavior with the supplied BERTopic, UMAP, HDBSCAN, and nr_topics=96 configuration, then inspect the topic-reduction stage reached by fit_transform. Determine whether the log reflects expected reduction semantics or an incorrect count, and document or test the observed behavior once the expected result is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100