MaartenGr / MaartenGr/BERTopic

Probabilities from fit_transform and transform are different

Open
#1,831 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Using bertopic==0.16.0 on a MacOS M1 machine, I have found some very strange behavior for the probabilities for each topic.

```
dataset = load_dataset("CShorten/ML-ArXiv-Papers")["train"]
docs = dataset["abstract"][:5000]
bertopic = BERTopic(
embedding_model=SentenceTransformer("all-MiniLM-L6-v2"),
calculate_probabilities=True
)
embeddings = bertopic.embedding_model.encode(docs, show_progress_bar=True)
y_pred, y_prob = bertopic.fit_transform(docs, embeddings)
y_pred_transform, y_prob_transform = bertopic.transform(docs, embeddings)
```

First of all, probabilities don't add up to 100% (addressed in #500), apparently because it does not account for not belonging ot any topic, which I guess is fine.
```
y_prob.sum(axis=1) --> [0.458, 0.386, 0.607, ..., 1., 0.927, 0.350]
```

However, I have noticed that `fit_transform` and `transform` don't return the same probabilities, which is obviously a concern. Which one should we trust then? Weirdly the sum of the probabilities are the same with both methods. The discrepancy between the 2 methods comes from the fact that `fit_transform` and `transform` are defined independently and do different things (typically, when using the sklearn API format, one defines the `fit` and `transform` methods and gets the `fit_transform` method for free, which ensures consistency between `fit_transform` and `transform` predictions).
```
np.allclose(y_pred, y_pred_transform) --> True
np.allclose(y_prob, y_prob_transform) --> False
np.allclose(y_prob.sum(axis=1), y_prob_transform.sum(axis=1)) --> True
```

What's even more surprising is that the predicted topic is not the topic with the highest probability. Another issue (#1024) raised this issue but it seemed to have been corrected in `v0.14.1`. The values only match for 71% of the documents in my case:
```
np.allclose(y_pred, np.argmax(y_prob, axis=1)) --> False
np.allclose(y_pred_transform, np.argmax(y_prob_transform, axis=1)) --> False
y_pred --> [70, 52, 6, ..., 1, -1, -1]
np.argmax(y_prob, axis=1) --> [65, 52, 6, ..., 1, 75, 12]
(np.argmax(y_prob, axis=1) == np.array(y_pred)).mean() --> 0.7128
```

So, given all these inconsistencies, should we trust probabilities at all?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied BERTopic 0.16.0 example and compare the outputs of fit_transform and transform, including their probability sums and predicted topics. Inspect the fit_transform and transform entry points to determine why their probabilities differ and why argmax does not match the predicted topic. Done means the expected probability semantics and consistency between both methods are established, with regression coverage if a defect is confirmed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.