Request: Zeroshot option to assign unassigned documents to outliers rather than reclustering

Nobody has claimed this yet.

Hi Maarten,

I have a use case at the moment where I'm using zero shot topic modeling to assign documents to a list of known clusters. I'm not really interested in finding other unknown clusters in the data, but I do know that there will be some documents that don't match anything and I would just like them to be outliers.

At the moment, the workflow for zeroshot is that any documents that dont match a zeroshot topic to a certain threshold go into a pool to be run through the standard bertopic pipeline. That's useful in some cases, but not others. One issue that I have encountered is that if there are only a few docs that don't fit into a topic (e.g. 4), UMAP can't handle it and produces an error (the same error in #1900 when I was trying to visualise only 4 topics).

Could we have an option in zeroshot to determine where to direct documents that fall below `zeroshot_min_similarity`? Either to outliers or to reclustering?

Cheers

Start from the zero-shot workflow and the `zeroshot_min_similarity` threshold described in the issue. Trace how below-threshold documents are currently sent to the standard BERTopic pipeline, then define and test an option that routes them either to outliers or to reclustering while avoiding the reported UMAP failure for very small pools.

Written by the indexing model from the issue text.