Revisiting HDBSCAN tuning and topic clustering

Nobody has claimed this yet.

This is a long post and I apologize going in. I also want to make clear that none of this should be read as a criticism of BERTopic or the choices made of where the product has focused its attention. In fact, after a year of using the software and having taken a deep dive to look at alternatives, I'm convinced that BERTopic is a unique project and the only end-to-end topic modeling package available. In particular, as I've mentioned before, the approach of clustering embeddings and the c-TF-IDF is really unique and powerful. That being said I think that it is worth going back and re-visiting some of the assumptions about the clustering and outliers.

I have poked around these issues in #556 where I commented about the one-way/two-way relationship between the embedded data and the actual topic model. This may sound relatively abstract - but we see it time and time again when users ask questions about how to reduce -1 categorizations and also how to get better individual document assignments from a fit model. To inform this issue I've prepared a notebook and data that I hope clearly steps through a case that is representative and compelling.

https://github.com/drob-xx/Tune_BERTopic_HDBSCAN

I have tried to make this as lightweight and usable as possible. I hope people have time to take a look.

Essentially my thesis is this:

- Without explicitly exploring and setting HDBSCAN for at least min_samples and min_cluster_size is a complete crapshoot. Virtually *any* selection without testing will result in extremely sub-optimal settings for *any* dataset.
- While many datasets should be thought of as having many 'outliers', there is also an argument to be made that there are no 'outliers' just some datasets that are on a spectrum from homogenous, cohesive, coherent to heterogeneous, incoherent and chaotic.
- From what I'm seeing, assuming a reasonably coherent set of data - in my example above a set of English language general news articles of like sized length and from a small number of reputable, professional news outlets, the number of 'outliers' should be minimal and the overall coherence quite high. While I realize this may not be the case - I would argue that if topic modeling is a valid pursuit - these are the kind of results we want.

I think I well understand the overall approach that BERTopic is taking. It is entirely reasonable and in many cases more than sufficient. If I were to summarize I would say that BERTopic is optimized to have a good out-of-box experience and that the clustering only has to be 'good-enough' to get a reasonable number of cluster members on top of which c-TF-IDF can run and produce a good set of topic vocabularies. In this view, what the job is to get good enough clusters. However, there are downsides to this approach (but of course software is always about trade-offs).

I've spent a good amount of time in this area. I think that it *might* be possible to build a light-weight configurator/optimizer which would help users get a better out-of-box experience, and which would certainly help users wanting to reduce outliers, fit their data better and more closely associate the model to their data. It is entirely possible that I've missed something, mis-understand what is going on or am simply off on a tangent. I'm very open to constructive criticism as my only interest is in becoming a better practitioner.

I will close with a couple of charts which illustrate just how difficult it is to select reasonable parameters for a given dataset without testing. These charts are relatively dense for those unfamiliar with the format - but I think are worth paying attention to. I'm happy to explain anything here or answer questions.

This first chart show the result of running HDBSCAN against the umap reductions for the data in my example. All the BERTopic and model settings are default and the value that is changing is 'min_cluster_size'. This is the result of 62 randomized values between five and six-hundred.

We can see that there are what I call 'natural' clusters bands: of under 10, then a group between 70 and 160, and one at around 280. We can also see that there is real variation in the number of outliers created.


Zooming in on the bottom grouping we see:


There is no clear relationship between the min_cluster_size and the actual number of clusters identified. While in general the larger number produce smaller sizes, There are lots of cases where this is not true - no way to know unless you run a bunch of tests.

I will further complicate by adding in the effect of min_samples which further complicates the picture. This shows min_samples size (as expressed here as a percentage of min_cluster_size) having a significant, and difficult to predict, effect on the output:


All of this may seem overwhelming. Also, from what I've learned the nature of topic modeling is quite perplexing. On the one hand it sits at the crux of machine learning and human cognition in a way that is very visceral. There is something about categorizing documents into human understandable that seems to me to be very powerful. On the other hand when one dives deeply into older technologies like LDA, it becomes quickly apparent that while useful, the techniques are very flawed. Within the industry, as far as I can tell, topic modeling is somewhat niche. The lack of ground-truth measures and the difficulty of easily measuring performance has understandably dampened interest. It seems to me that modeling based on the embeddings is much, much more powerful than the older approaches and deserves more attention. Please see my scatter plots in the notebook showing how logically documents are placed in relation to one another based on the embeddings. It seems that the trick is getting the clustering algorithm to 'see' the patterns, and as I point out in my notebook, it is doable, but requires some work.

I'm just getting started in this area and hope that this is interesting and not a digression.

Start with the linked Tune_BERTopic_HDBSCAN notebook and the discussion in issue #556. Review how BERTopic currently exposes HDBSCAN settings, then determine whether the proposed configurator or optimizer has a defined scope and evaluation method. Done would require an agreed design and measurable validation criteria, which this issue does not yet specify.

Written by the indexing model from the issue text.