MaartenGr / MaartenGr/BERTopic

How to get top N topics for a single document.

Open
#1,398 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

I want to ask some simple questions.
After fitting, transforming, reducing the outliers, and updating the topic model this way:

```
embedding = ...
topics, probs = model.fit_transform(docs, embeddings = emb)
new_topics = model.reduce_outliers(docs, topics)
model.update_topics(new_docs, topics=new_topics)
```
I want to predict one single new document this way:
`prediction= model.transform(documents=new_document, embeddings=embedding)`
The result is the following:
``([-1],
array([[1.16350449e-02, 2.01218509e-01, 1.11319454e-02, 7.07114037e-04,
3.97837834e-03, 2.08542765e-04, 3.01921767e-03, 7.32074922e-02,
4.58492993e-01, 9.87979027e-03, 9.41436941e-03, 1.09079184e-02,
1.02977729e-02, 9.64706333e-03, 1.12269956e-02, 1.04989969e-02,
3.24051104e-03, 1.23429396e-02, 1.00992529e-02, 9.99152714e-03,
1.06927620e-02, 1.20518505e-02, 1.06281160e-02, 4.32279154e-03]]))``

The tuple contains:

- The topic prediction [-1]
- An array of probabilities for the other remaining topics

Am I right?

Why is the topic prediction [-1] when I expressly removed the outliers and updated the model?
If I simply call `model.topic_labels_ ` I don't get the [-1] topic anymore and it starts from [0] as expected.
Why is my model still predicting the topic [-1]?

Moreover, I guess the array of probabilities is following the normal ascending order (in my case from 0 to 23). Right?
If that's the case I can imagine that the 8th value in this list [4.58492993e-01] (9th if counting non-pythonically) is the one corresponding to the most corresponding topic. Is this true?

I am asking you this so that I can easily predict the top N topics for a single new document.

I have reviewed the BERTopic documentation and explored the available methods, but I couldn't find a straightforward way to obtain the probabilities for all topics beyond the top 1 prediction for a single new document.

Could you please help me? I would greatly appreciate any assistance or suggested workarounds for achieving this.

Thank you in advance!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the documented fit_transform, reduce_outliers, update_topics, transform, and topic_labels_ entry points, then reproduce the reported return values with a single document. Done means determining whether the [-1] topic and probability ordering are expected and identifying a documented way to obtain the top N topics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.