MaartenGr / MaartenGr/BERTopic
Lightweight soft topic assignment via temperature-scaled embedding distances
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Feature request
Add a `soft_clustering_temp` parameter to `transform()` that computes soft topic distributions using temperature-scaled softmax on embedding distances to topic centroids:
```python
# Hard assignment (current default)
topics, probs = topic_model.transform(new_docs)
# Soft assignment with temperature scaling
topics, probs = topic_model.transform(new_docs, soft_clustering_temp=0.5)
# probs is now a (n_docs, n_topics) matrix of soft assignments
```
### Motivation
Getting soft (probabilistic) topic assignments currently requires either:
1. `calculate_probabilities=True` — expensive for large datasets and HDBSCAN-specific
2. `approximate_distribution()` — good but requires re-running the vectorizer
Users frequently ask about soft/overlapping topic assignments:
- [#1419](https://github.com/MaartenGr/BERTopic/issues/1419) — non-mutually exclusive topics
- [#2017](https://github.com/MaartenGr/BERTopic/issues/2017) — probability inconsistency
- [#1613](https://github.com/MaartenGr/BERTopic/issues/1613) — negative probabilities
- [#1808](https://github.com/MaartenGr/BERTopic/issues/1808) — probabilities NoneType
- [#1962](https://github.com/MaartenGr/BERTopic/issues/1962) — probabilities empty with zero-shot
The temperature-scaled approach is clustering-agnostic (works with any clustering model, not just HDBSCAN), lightweight (no vectorizer, no refit), and provides a smooth probability distribution over all topics.
### Your contribution
I can submit a PR that adds `soft_clustering_temp` to `transform()`. Low temperature → sharper (near-hard) assignments; high temperature → softer distributions. Default is `None` — existing behavior unchanged.
I've already been prototyping this in my fork (working implementation with tests). Since this adds a new parameter, I'd value your steer on the API (name, whether it lives on `transform()` or a dedicated method) before I open the PR.
---
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The requested entry point is transform(); start by locating its implementation and existing tests, then review how current topic assignments and probabilities are produced. Confirm the API and temperature behavior with maintainers before implementing; done means the default remains unchanged and the new option returns tested soft assignments across supported clustering models.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100