MaartenGr / MaartenGr/BERTopic
Problem with model.transform() after Outlier reduction procedure
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hello,
I am trying to extract topics from a list of texts. Since my data probably lacks a bit of quality and half of the texts were classified as outliers, I proceeded with the outlier reduction stage after the topic extraction one.
This is the code:
```
self.model = BERTopic(
language="italian",
top_n_words=10,
n_gram_range=(1, 1),
min_topic_size=40,
nr_topics="auto",
embedding_model=embedding_model,
seed_topic_list=seed_topic_list,
calculate_probabilities=False,
# verbose=self.verbose
)
# Fit the model to the documents
topics, _ = self.model.fit_transform(self.history_tickets)
# Use this method to reduce the number of outliers taken, and get the new topics
new_topics = self.model.reduce_outliers(self.history_tickets, topics, strategy="c-tf-idf")
# Then, update the topics to the ones that considered the new data
self.model.update_topics(self.history_tickets, topics=new_topics)
```
And it works fine. However, when I proceed with the extraction of the topic of a single text (that was already part of self.history_tickets):
`prediction = self.model.transform([text])[0][0]`
the prediction is in most cases -1.
What is the problem? Should I proceed with reduce_outliers also after the single prediction?
Thanks in advance,
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the shown BERTopic fit_transform, reduce_outliers, update_topics, and transform sequence. Read the behavior of reduce_outliers and update_topics alongside transform, then verify whether transforming an already-seen text should return its updated topic rather than -1. Done means the expected behavior is confirmed and covered by a regression test or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100