MaartenGr / MaartenGr/BERTopic

Supervised Topic Modelling unable to produce same output using fit_transform() and transform()

Open
#1,270 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi,
I referenced the code from https://maartengr.github.io/BERTopic/getting_started/supervised/supervised.html for BERTopic Supervised modelling. Although I was able to obtain the correct output using fit_transform(docs, y=y), I was unable to obtain the same output using transform(docs) even though the inputted doc is the same. Most of the topics generated were labelled -1.

May I know what has gone wrong?

For reference, this is my model code:
empty_dimensionality_model = BaseDimensionalityReduction()
clf = LogisticRegression()
ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True)

topic_model = BERTopic(
umap_model=empty_dimensionality_model,
hdbscan_model=clf,
ctfidf_model=ctfidf_model,
n_gram_range=(1,3))

Thank you! Any help would be truly appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supervised BERTopic example linked in the issue and compare the shown fit_transform(docs, y=y) and transform(docs) calls using the provided model configuration. Investigate why transform assigns many topics as -1; done means the behavior is explained and the same input produces consistent topic assignments, with a regression test if the repository provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.