MaartenGr / MaartenGr/BERTopic
Representative docs requests may fail after multiple rounds of partial_fit in online topic modelling
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi,
I'm using the online topic modelling with River, calling like this (eg for a list of 200 docs)
first = docs[:100]
second = docs[100:]
self.topic_model.partial_fit(first)
self.topic_model._save_representative_docs(docs)
self.topic_model.partial_fit(second)
self.topic_model._save_representative_docs(docs)
After the second call I want to check if new clusters were generated by River, and if so retrieve the representative docs for the new clusters. BERtopic does not take this action after partial_fit, so I am running it manually. However, running the save method the second time results in an exception:
if ensure_min_samples > 0:
n_samples = _num_samples(array)
if n_samples < ensure_min_samples:
> raise ValueError(
"Found array with %d sample(s) (shape=%s) while a"
" minimum of %d is required%s."
% (n_samples, array.shape, ensure_min_samples, context)
)
E ValueError: Found array with 0 sample(s) (shape=(0, 1018)) while a minimum of 1 is required by the normalize function.
.venv/lib/python3.11/site-packages/sklearn/utils/validation.py:967: ValueError
This looks to be caused because there are zero samples in a cluster that was generated in the first partial_fit call in the second set of docs. In this case it would be great if the library skipped the empty cluster, and just emitted the representative docs it does have.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the _save_representative_docs entry point and reproduce the two partial_fit calls shown in the issue. Check how representative documents are selected when a cluster has zero samples; done means the second save completes and returns representative documents for non-empty clusters without the normalization exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100