MaartenGr / MaartenGr/BERTopic

Probabilities are NoneType when doing supervised learning

Open
#1,808 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

I'm following the guide [here](https://maartengr.github.io/BERTopic/getting_started/supervised/supervised.html) on how to use BERTopic for supervised learning on the `20newsgroups` dataset. I want to see all calculated probabilities so I added `calculate_probabilities=True` to the BERTopic instance, see below. However, `probs` comes up as `NoneType`. Is some dependency I am lacking or something wrong with my implementation?

`probs` does return an array of probabilities when doing unsupervised learning, so I wonder if this issue is unique to supervised learning.

```
# Skip over dimensionality reduction, replace cluster model with classifier,
# and reduce frequent words while we are at it.
empty_dimensionality_model = BaseDimensionalityReduction()
clf = LogisticRegression()
ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True)

# Create a fully supervised BERTopic instance
topic_model= BERTopic(
umap_model=empty_dimensionality_model,
hdbscan_model=clf,
ctfidf_model=ctfidf_model,
calculate_probabilities = True
)

topics, probs = topic_model.fit_transform(docs, y=y)
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the supervised-learning example from the linked guide with the 20newsgroups dataset, LogisticRegression, and calculate_probabilities=True. Trace the supervised fit_transform path to determine why probs is None while unsupervised learning returns an array. Done means the behavior is fixed or the supervised limitation is clearly documented, with the reproduction covered by a regression test if the project provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.