MaartenGr / MaartenGr/BERTopic

Zero-shot topics are merged during topic reduction

Open
#2,409 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

### Have you searched existing issues? 🔎

- [x] I have searched and found no existing issues

### Desribe the bug

Thanks for this excellent library! I've been experimenting with zero-shot topic modeling and encountered some (to me) unexpected behavior. I'm unsure whether this is intended functionality or a bug.

**Setup:**
I am running a basic BERTopic model as follows:
```
topic_model = BERTopic(
embedding_model = SentenceTransformer(sentence_transformer_dir),
min_topic_size=10,
language="multilingual",
nr_topics="auto",
n_gram_range=(1, 2),
zeroshot_topic_list=zeroshot_topic_list,
zeroshot_min_similarity=.50,
)
```
with a basic `zeroshot_topic_list` looking something like:
```
zeroshot_topic_list = [
"machine battery is empty",
"machine is broken does not work",
"I want a new machine",
]
```

**Issue:**
Despite having documents that clearly match my zero-shot topics, I frequently observed that either:
1. Zero-shot topics are completely missing from the final results;
2. When zero-shot topics do appear, they contain documents that would better match other zero-shot topics in the list (making the document count in this topic higher than expected).

**Investigation:**
I investigated the issue by examining this code section:
https://github.com/MaartenGr/BERTopic/blob/32b2dddea3df57e9982b166e5b1847c9fbe7e6da/bertopic/_bertopic.py#L4018-L4023

At this stage, as I expected, all three zero-shot topics would get assigned documents. After running `assigned_documents["Topic"].value_counts()` my output is as follows:
```
Topic
1 1506
2 1426
0 1183
Name: count, dtype: int64
```

Based on my understanding of the documentation, zero-shot topic modeling should:
1. First assign zero-shot topics to documents, using the specified cosine similarity thresholds.
2. Run the remainder of the topic-modeling pipeline only on the documents that have not been assigned to any of the zero-shot topics.
3. **Exclude zero-shot topics from any subsequent topic reduction operations.**

The behavior described in the last point is what I'm most uncertain about.

I want zero-shot topics to remain untouched after initial assignment, with the remaining topic modeling pipeline applied only to unassigned documents. Topic reduction should affect only the dynamically discovered topics, not the predefined zero-shot topics.

The behavior appears to come from zero-shot topics being included in the `_reduce_topic` call here:
https://github.com/MaartenGr/BERTopic/blob/32b2dddea3df57e9982b166e5b1847c9fbe7e6da/bertopic/_bertopic.py#L514-L520
Is this intended behavior?

When I disable topic reduction by setting `nr_topics=None`, all zero-shot topics appear correctly with expected document assignments. After running `topic_model.get_topic_info()`, my output is as follows:
```
Topic | Count | Name
-- | -- | --
-1 | |
0 | 1183 | machine battery is empty
1 | 1506 | machine is broken does not work
2 | 1426 | I want a new machine
```
However, when setting `nr_topics=None`, no topic reduction is being performed on the dynamically discovered topics.

**Question:**
Is the current behavior where zero-shot topics are subject to topic reduction intentional? If so, would it be possible to add an option to preserve zero-shot topics during the reduction phase?

### Reproduction

_No response_

### BERTopic Version

0.17.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in bertopic/_bertopic.py at the linked zero-shot assignment section and the _reduce_topic call. Trace how zero-shot topic IDs enter topic reduction, then compare the behavior with nr_topics=None. Done means zero-shot topics retain their assignments while reduction still applies to dynamically discovered topics, with the reported behavior covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.