MaartenGr / MaartenGr/BERTopic
How to get top N topics for a single document.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
I want to ask some simple questions.
After fitting, transforming, reducing the outliers, and updating the topic model this way:
```
embedding = ...
topics, probs = model.fit_transform(docs, embeddings = emb)
new_topics = model.reduce_outliers(docs, topics)
model.update_topics(new_docs, topics=new_topics)
```
I want to predict one single new document this way:
`prediction= model.transform(documents=new_document, embeddings=embedding)`
The result is the following:
``([-1],
array([[1.16350449e-02, 2.01218509e-01, 1.11319454e-02, 7.07114037e-04,
3.97837834e-03, 2.08542765e-04, 3.01921767e-03, 7.32074922e-02,
4.58492993e-01, 9.87979027e-03, 9.41436941e-03, 1.09079184e-02,
1.02977729e-02, 9.64706333e-03, 1.12269956e-02, 1.04989969e-02,
3.24051104e-03, 1.23429396e-02, 1.00992529e-02, 9.99152714e-03,
1.06927620e-02, 1.20518505e-02, 1.06281160e-02, 4.32279154e-03]]))``
The tuple contains:
- The topic prediction [-1]
- An array of probabilities for the other remaining topics
Am I right?
Why is the topic prediction [-1] when I expressly removed the outliers and updated the model?
If I simply call `model.topic_labels_ ` I don't get the [-1] topic anymore and it starts from [0] as expected.
Why is my model still predicting the topic [-1]?
Moreover, I guess the array of probabilities is following the normal ascending order (in my case from 0 to 23). Right?
If that's the case I can imagine that the 8th value in this list [4.58492993e-01] (9th if counting non-pythonically) is the one corresponding to the most corresponding topic. Is this true?
I am asking you this so that I can easily predict the top N topics for a single new document.
I have reviewed the BERTopic documentation and explored the available methods, but I couldn't find a straightforward way to obtain the probabilities for all topics beyond the top 1 prediction for a single new document.
Could you please help me? I would greatly appreciate any assistance or suggested workarounds for achieving this.
Thank you in advance!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the documented fit_transform, reduce_outliers, update_topics, transform, and topic_labels_ entry points, then reproduce the reported return values with a single document. Done means determining whether the [-1] topic and probability ordering are expected and identifying a documented way to obtain the top N topics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100