MaartenGr / MaartenGr/BERTopic

Calculating Probabilities on Zero-Shot Learning

Open
#2,076 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement good first issue
Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Have you searched existing issues? 🔎

- [X] I have searched and found no existing issues

### Desribe the bug

Hi everyone, first of all I would like to thank @MaartenGr and all the contributors for this amazing project.

Recently I started looking at BERTopic as a method to classify some customer tickets into some categories defined into`zeroshot_topic_list` parameter. After fitting the model by calling `fit_transform` my goal was to look for each document, the probability of that document of belonging to all the topics (both predefined and generated ones).

`probs` is `None` after `fit_transform`, as expected and mentioned at the end of this page https://maartengr.github.io/BERTopic/getting_started/zeroshot/zeroshot.html#example. Therefore, I then called `transform` to get the probabilities.

Now I've got two questions I would like to answer:
- is this one the right approach to get the probabilities?
- most of the documents have almost the same (high) probability among all the topics. Does this mean the clustering didn't fit that much the data? What do you suggest?

Again, I would like to thank you for your effort in advance and I look forward to contribute as well if needed.

### Reproduction

```python
topic_model = BERTopic(
calculate_probabilities = True,
vectorizer_model = CountVectorizer(stop_words=default_stopwords + custom_stopwords),
ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True),
embedding_model = sentence_model,
min_topic_size = 50,
zeroshot_topic_list = taxonomy_list,
zeroshot_min_similarity = .80,
representation_model = KeyBERTInspired(),
verbose = True,
)

topics, _ = topic_model.fit_transform(global_ticket_descriptions, embeddings=embeddings)
_, probs = topic_model.transform(global_ticket_descriptions, embeddings=embeddings)
print(probs)
```

The output is:
```
2024-07-09 14:48:15,691 - BERTopic - Predicting topic assignments through cosine similarity of topic and document embeddings.
[[0.8463203 0.8880229 0.8636132 ... 0.8376566 0.8344382 0.82069004]
[0.92492086 0.8977871 0.91309047 ... 0.9031642 0.90693504 0.9101905 ]
[0.9009018 0.9072951 0.91581297 ... 0.9002963 0.8903491 0.8731015 ]
...
[0.85506856 0.9055494 0.8743948 ... 0.8490986 0.8607102 0.8375366 ]
[0.8783543 0.88571143 0.8983458 ... 0.8795974 0.87251174 0.8653137 ]
[0.8480984 0.8751351 0.8680049 ... 0.8383051 0.8333502 0.8458183 ]]
```

### BERTopic Version

0.16.2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report exercises BERTopic.fit_transform and transform with calculate_probabilities=True and zeroshot_topic_list; begin by tracing those entry points and the zero-shot probability handling. No file or test is named, and completion criteria are not defined, so first confirm whether this is a bug requiring a code or test change or documentation explaining the observed probabilities.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.