MaartenGr / MaartenGr/BERTopic

Problem with model.transform() after Outlier reduction procedure

Open
#1,507 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hello,
I am trying to extract topics from a list of texts. Since my data probably lacks a bit of quality and half of the texts were classified as outliers, I proceeded with the outlier reduction stage after the topic extraction one.
This is the code:

```
self.model = BERTopic(
language="italian",
top_n_words=10,
n_gram_range=(1, 1),
min_topic_size=40,
nr_topics="auto",
embedding_model=embedding_model,
seed_topic_list=seed_topic_list,
calculate_probabilities=False,
# verbose=self.verbose
)
# Fit the model to the documents
topics, _ = self.model.fit_transform(self.history_tickets)

# Use this method to reduce the number of outliers taken, and get the new topics
new_topics = self.model.reduce_outliers(self.history_tickets, topics, strategy="c-tf-idf")
# Then, update the topics to the ones that considered the new data
self.model.update_topics(self.history_tickets, topics=new_topics)
```

And it works fine. However, when I proceed with the extraction of the topic of a single text (that was already part of self.history_tickets):

`prediction = self.model.transform([text])[0][0]`

the prediction is in most cases -1.
What is the problem? Should I proceed with reduce_outliers also after the single prediction?

Thanks in advance,

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the shown BERTopic fit_transform, reduce_outliers, update_topics, and transform sequence. Read the behavior of reduce_outliers and update_topics alongside transform, then verify whether transforming an already-seen text should return its updated topic rather than -1. Done means the expected behavior is confirmed and covered by a regression test or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.