Potential shortfall/optimisation of data flow process

Nobody has claimed this yet.

I thought I was done giving helpful feedback, but I just had a really interesting clustering problem and the implications of the solution directly impacts your product. I cannot deny that the data flow process you use is powerful and effective, so I am not criticising that.

However, I just had a case of highly similar data and the only way that I could cluster it and visualise the clusters was to cluster it in 50 dimensions and then colour the reduced dimensions according to those clusters. This produced a highly concentric clustering diagram that looked like an eye surrounded by freckles.

The shortfall was that this was difficult to do/troubleshoot with BERTopic, but the optimisation I propose is to allow the user to pass in an iterable, with a list of operations and then BERTopic follows the custom workflow.

One of my use cases for BERTopic is to create summaries of thousands of news articles. To do this I need to run my data through BERTopic three times with different settings. Passing in a list for vanilla function, or a dictionary with key=process, value=(dict with kwargs for process or None), would allow a user to specify a workflow so that BERTopic only needs to be called once.

But as it is, this particular dataset could not be clustered successfully in lower dimensions as it was too spherical/self-similar. I do not expect this to be uncommon in the domain of news topics, where certain things are covered mostly in one light or for a limited audience.

PS. You can close the issue with a reply, no point keeping it open once you have read it.

Start at BERTopic's current public workflow and compare it with the issue's three-pass news-article use case. Define the desired sequence of operations and its configuration shape before implementation; done should mean the workflow can be specified once and applied consistently, including the high-dimensional clustering case described.

Written by the indexing model from the issue text.