'Reduce Outliers' does not work when using zero shot topics

Nobody has claimed this yet.

Hi Maarten - first, many thanks for the amazing library and continued updates! Just wanted to point out that when using zero shot topics, the outlier topic ID changes from -1 to some positive integer depending on however many zero shot topics the model accepts. This causes issues with the 'reduce outliers' function which looks for outliers using topic ID == -1. Please see attached screenshot. In this case, my outliers have a topic ID of '4' instead of '-1' because of which the reduce outliers function is not able to identify them. Thanks!


This is the code snippet from the reduce outliers function:

# Reduce outliers by finding the most similar topic embeddings
elif strategy.lower() == "embeddings":
if self.embedding_model is None and embeddings is None:
raise ValueError("To use this strategy, you will need to pass a model to `embedding_model`"
"when instantiating BERTopic.")
outlier_ids = [index for index, topic in enumerate(topics) if topic == -1]

Start at the reduce outliers function, especially the embeddings strategy shown in the issue, and trace how zero-shot topic IDs are assigned. Reproduce the case where outliers receive a positive ID, then verify that the reduce outliers behavior identifies and processes those documents correctly.

Written by the indexing model from the issue text.