Reducing outliers by clustering them

Nobody has claimed this yet.

This is a question, more than an issue. If it should be on SO, let me know and i will post there. Apologies in advance if that is the case.

I have a dataset of social media content that is producing reasonable clusters. When I look at the clusters over time (not using the built-in method, instead summing topic numbers by month of original post) i see some sensible spikes in topics. There is a period, however when i see a number of clusters all spiking, including outliers. Perhaps this is just a period of increased social media usage, but i suspect there is at least one latently valuable cluster in my outlier group.

What i would like to do is try and reduce the number of outliers by simply running another topic model, just on the outlier data. When i do this i come up with five new topics, three of which appear to see increased activity under the period of increased activity i just described.

I know this isn't covered among the approaches to reducing outliers, and i am wondering if its a viable method, or if i am just generating garbage or incommensurate data. My hope would be to subsequently manually merge the outlier and base models, so i end up with all the topics from the base model, plus the five new topics from the outlier model, with the outliers now reduced to only those that remain in the outlier model? I have seen you suggest something similar for achieving greater granularity in large clusters, but i could not find anything about the validity of doing this with the outliers. To my mind it should work, but perhaps i have to break the analysis out into the two separate models.

For what its worth, i tried the built in outlier reduction method, and that successfully allocated the outliers to existing topics. However that just made my topics make less sense given the diversity of content in them. I also tried reducing the minimum cluster size, but that gave me less useful results.

Finally, i should note that when i run the outlier model, I am using the same hyperparameters as the base model.

Thanks for all your amazing work on this package.

The issue names no files, tests, or entry points. Start by reviewing BERTopic's built-in outlier reduction method and the separate outlier model described in the report. Done would require a maintainer decision on whether this modeling approach is supported, rather than an isolated code change.

Written by the indexing model from the issue text.