MaartenGr / MaartenGr/BERTopic
Probabilities are NoneType when doing supervised learning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
I'm following the guide [here](https://maartengr.github.io/BERTopic/getting_started/supervised/supervised.html) on how to use BERTopic for supervised learning on the `20newsgroups` dataset. I want to see all calculated probabilities so I added `calculate_probabilities=True` to the BERTopic instance, see below. However, `probs` comes up as `NoneType`. Is some dependency I am lacking or something wrong with my implementation?
`probs` does return an array of probabilities when doing unsupervised learning, so I wonder if this issue is unique to supervised learning.
```
# Skip over dimensionality reduction, replace cluster model with classifier,
# and reduce frequent words while we are at it.
empty_dimensionality_model = BaseDimensionalityReduction()
clf = LogisticRegression()
ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True)
# Create a fully supervised BERTopic instance
topic_model= BERTopic(
umap_model=empty_dimensionality_model,
hdbscan_model=clf,
ctfidf_model=ctfidf_model,
calculate_probabilities = True
)
topics, probs = topic_model.fit_transform(docs, y=y)
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the supervised-learning example from the linked guide with the 20newsgroups dataset, LogisticRegression, and calculate_probabilities=True. Trace the supervised fit_transform path to determine why probs is None while unsupervised learning returns an array. Done means the behavior is fixed or the supervised limitation is clearly documented, with the reproduction covered by a regression test if the project provides one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100