MaartenGr / MaartenGr/BERTopic
Reducing Outliers of Loaded Model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi!
A month ago I created a topic model and saved it as follows: `topic_model.save(outpath, serialization="safetensors")`.
I then reduced the outliers in the model, `new_topics = topic_model.reduce_outliers(docs, topics)`, and used it in an empirical analysis, but I did not save the model with the updated topics.
I now want to produce visualizations of the topics used in the analysis so I have loaded my dataframe (and defined docs again), loaded the model and tried to reduce the outliers again, but I get an error and I am not sure how to fix it. The code and error are below:
```
loaded_model = BERTopic.load("Only-English-BERT-topic-meaning-min-size-50")
topics = loaded_model.topics_
new_topics = loaded_model.reduce_outliers(docs, topics)
```
_sklearn.exceptions.NotFittedError: Vocabulary not fitted or provided__
I have also tried using
> `topics, probs = loaded_model.transform(docs)
, but I got the same error.
Any help in how to fix this would be greatly appreciated.
Thanks in advance for your time!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing BERTopic.load and the reduce_outliers and transform entry points described in the report, then reproduce the NotFittedError with the saved safetensors model. Check whether loading preserves the vocabulary needed by these operations. Done means the loaded model can reduce outliers or transform the documents without the reported error, with the behavior covered by an appropriate regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100