MaartenGr / MaartenGr/BERTopic
Probabilities from fit_transform and transform are different
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Using bertopic==0.16.0 on a MacOS M1 machine, I have found some very strange behavior for the probabilities for each topic.
```
dataset = load_dataset("CShorten/ML-ArXiv-Papers")["train"]
docs = dataset["abstract"][:5000]
bertopic = BERTopic(
embedding_model=SentenceTransformer("all-MiniLM-L6-v2"),
calculate_probabilities=True
)
embeddings = bertopic.embedding_model.encode(docs, show_progress_bar=True)
y_pred, y_prob = bertopic.fit_transform(docs, embeddings)
y_pred_transform, y_prob_transform = bertopic.transform(docs, embeddings)
```
First of all, probabilities don't add up to 100% (addressed in #500), apparently because it does not account for not belonging ot any topic, which I guess is fine.
```
y_prob.sum(axis=1) --> [0.458, 0.386, 0.607, ..., 1., 0.927, 0.350]
```
However, I have noticed that `fit_transform` and `transform` don't return the same probabilities, which is obviously a concern. Which one should we trust then? Weirdly the sum of the probabilities are the same with both methods. The discrepancy between the 2 methods comes from the fact that `fit_transform` and `transform` are defined independently and do different things (typically, when using the sklearn API format, one defines the `fit` and `transform` methods and gets the `fit_transform` method for free, which ensures consistency between `fit_transform` and `transform` predictions).
```
np.allclose(y_pred, y_pred_transform) --> True
np.allclose(y_prob, y_prob_transform) --> False
np.allclose(y_prob.sum(axis=1), y_prob_transform.sum(axis=1)) --> True
```
What's even more surprising is that the predicted topic is not the topic with the highest probability. Another issue (#1024) raised this issue but it seemed to have been corrected in `v0.14.1`. The values only match for 71% of the documents in my case:
```
np.allclose(y_pred, np.argmax(y_prob, axis=1)) --> False
np.allclose(y_pred_transform, np.argmax(y_prob_transform, axis=1)) --> False
y_pred --> [70, 52, 6, ..., 1, -1, -1]
np.argmax(y_prob, axis=1) --> [65, 52, 6, ..., 1, 75, 12]
(np.argmax(y_prob, axis=1) == np.array(y_pred)).mean() --> 0.7128
```
So, given all these inconsistencies, should we trust probabilities at all?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied BERTopic 0.16.0 example and compare the outputs of fit_transform and transform, including their probability sums and predicted topics. Inspect the fit_transform and transform entry points to determine why their probabilities differ and why argmax does not match the predicted topic. Done means the expected probability semantics and consistency between both methods are established, with regression coverage if a defect is confirmed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100