MaartenGr / MaartenGr/BERTopic
Supervised Topic Modelling unable to produce same output using fit_transform() and transform()
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi,
I referenced the code from https://maartengr.github.io/BERTopic/getting_started/supervised/supervised.html for BERTopic Supervised modelling. Although I was able to obtain the correct output using fit_transform(docs, y=y), I was unable to obtain the same output using transform(docs) even though the inputted doc is the same. Most of the topics generated were labelled -1.
May I know what has gone wrong?
For reference, this is my model code:
empty_dimensionality_model = BaseDimensionalityReduction()
clf = LogisticRegression()
ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True)
topic_model = BERTopic(
umap_model=empty_dimensionality_model,
hdbscan_model=clf,
ctfidf_model=ctfidf_model,
n_gram_range=(1,3))
Thank you! Any help would be truly appreciated.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supervised BERTopic example linked in the issue and compare the shown fit_transform(docs, y=y) and transform(docs) calls using the provided model configuration. Investigate why transform assigns many topics as -1; done means the behavior is explained and the same input produces consistent topic assignments, with a regression test if the repository provides one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100