MaartenGr / MaartenGr/BERTopic
Zero-shot topics are merged during topic reduction
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Have you searched existing issues? 🔎
- [x] I have searched and found no existing issues
### Desribe the bug
Thanks for this excellent library! I've been experimenting with zero-shot topic modeling and encountered some (to me) unexpected behavior. I'm unsure whether this is intended functionality or a bug.
**Setup:**
I am running a basic BERTopic model as follows:
```
topic_model = BERTopic(
embedding_model = SentenceTransformer(sentence_transformer_dir),
min_topic_size=10,
language="multilingual",
nr_topics="auto",
n_gram_range=(1, 2),
zeroshot_topic_list=zeroshot_topic_list,
zeroshot_min_similarity=.50,
)
```
with a basic `zeroshot_topic_list` looking something like:
```
zeroshot_topic_list = [
"machine battery is empty",
"machine is broken does not work",
"I want a new machine",
]
```
**Issue:**
Despite having documents that clearly match my zero-shot topics, I frequently observed that either:
1. Zero-shot topics are completely missing from the final results;
2. When zero-shot topics do appear, they contain documents that would better match other zero-shot topics in the list (making the document count in this topic higher than expected).
**Investigation:**
I investigated the issue by examining this code section:
https://github.com/MaartenGr/BERTopic/blob/32b2dddea3df57e9982b166e5b1847c9fbe7e6da/bertopic/_bertopic.py#L4018-L4023
At this stage, as I expected, all three zero-shot topics would get assigned documents. After running `assigned_documents["Topic"].value_counts()` my output is as follows:
```
Topic
1 1506
2 1426
0 1183
Name: count, dtype: int64
```
Based on my understanding of the documentation, zero-shot topic modeling should:
1. First assign zero-shot topics to documents, using the specified cosine similarity thresholds.
2. Run the remainder of the topic-modeling pipeline only on the documents that have not been assigned to any of the zero-shot topics.
3. **Exclude zero-shot topics from any subsequent topic reduction operations.**
The behavior described in the last point is what I'm most uncertain about.
I want zero-shot topics to remain untouched after initial assignment, with the remaining topic modeling pipeline applied only to unassigned documents. Topic reduction should affect only the dynamically discovered topics, not the predefined zero-shot topics.
The behavior appears to come from zero-shot topics being included in the `_reduce_topic` call here:
https://github.com/MaartenGr/BERTopic/blob/32b2dddea3df57e9982b166e5b1847c9fbe7e6da/bertopic/_bertopic.py#L514-L520
Is this intended behavior?
When I disable topic reduction by setting `nr_topics=None`, all zero-shot topics appear correctly with expected document assignments. After running `topic_model.get_topic_info()`, my output is as follows:
```
Topic | Count | Name
-- | -- | --
-1 | |
0 | 1183 | machine battery is empty
1 | 1506 | machine is broken does not work
2 | 1426 | I want a new machine
```
However, when setting `nr_topics=None`, no topic reduction is being performed on the dynamically discovered topics.
**Question:**
Is the current behavior where zero-shot topics are subject to topic reduction intentional? If so, would it be possible to add an option to preserve zero-shot topics during the reduction phase?
### Reproduction
_No response_
### BERTopic Version
0.17.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in bertopic/_bertopic.py at the linked zero-shot assignment section and the _reduce_topic call. Trace how zero-shot topic IDs enter topic reduction, then compare the behavior with nr_topics=None. Done means zero-shot topics retain their assignments while reduction still applies to dynamically discovered topics, with the reported behavior covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100