MaartenGr / MaartenGr/BERTopic
Predicted probabilities inconsistency and questions about saving and loading model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi I have two questions when I use a HDBSCAN model with BERTopic.
-
When I try to predict a new sentence (I want the probability of the outlier as well so I am using HDBSCAN for prediction instead of approximiate_distribution function), I try to use topic_model.transform(text_input[0]) on my first training text but it gives me a different probability distribution from the probabilities probs[0] for the same sentence outputted by topics, probs = topic_model.fit_transform(text_input) when I fit the model/predict with the calculate_probabilities =True. Shouldn't they be the same as they are using the same trained HDBSCAN model for prediction? Why is this inconsistency from?
-
I have my own sentence transformer model (say that it is a fenetuned SBERT or wrapper of the BERT CLS token into sentence transformer class). If i want to keep my HDBSCAN and embedding model for inference when i reload my saved model, I can only save it as pickle format right? As per my understanding, the pytorch and safetensors saving options are only supporting huggingface factory sentence transformer (by re-downloading) instead of a customized or local saved sentence transformer and it will not save the HDBSCAN model, is that correct? Any lighter option (smaller saving size) for my purpose?
Thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported topic_model.fit_transform(text_input) and topic_model.transform(text_input[0]) calls, then compare how their probability outputs are produced. Review the model's save and load options for custom sentence-transformer and HDBSCAN components. Done means documenting the cause of the probability difference and the supported way to preserve these components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100