MaartenGr / MaartenGr/BERTopic

Reducing Outliers of Loaded Model

Open
#1,944 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi!

A month ago I created a topic model and saved it as follows: `topic_model.save(outpath, serialization="safetensors")`.

I then reduced the outliers in the model, `new_topics = topic_model.reduce_outliers(docs, topics)`, and used it in an empirical analysis, but I did not save the model with the updated topics.

I now want to produce visualizations of the topics used in the analysis so I have loaded my dataframe (and defined docs again), loaded the model and tried to reduce the outliers again, but I get an error and I am not sure how to fix it. The code and error are below:

```
loaded_model = BERTopic.load("Only-English-BERT-topic-meaning-min-size-50")
topics = loaded_model.topics_
new_topics = loaded_model.reduce_outliers(docs, topics)
```
_sklearn.exceptions.NotFittedError: Vocabulary not fitted or provided__

I have also tried using

> `topics, probs = loaded_model.transform(docs)

, but I got the same error.

Any help in how to fix this would be greatly appreciated.

Thanks in advance for your time!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing BERTopic.load and the reduce_outliers and transform entry points described in the report, then reproduce the NotFittedError with the saved safetensors model. Check whether loading preserves the vocabulary needed by these operations. Done means the loaded model can reduce outliers or transform the documents without the reported error, with the behavior covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.