MaartenGr / MaartenGr/BERTopic

Representative docs requests may fail after multiple rounds of partial_fit in online topic modelling

Open
#1,620 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hi,

I'm using the online topic modelling with River, calling like this (eg for a list of 200 docs)

first = docs[:100]
second = docs[100:]

self.topic_model.partial_fit(first)
self.topic_model._save_representative_docs(docs)

self.topic_model.partial_fit(second)
self.topic_model._save_representative_docs(docs)

After the second call I want to check if new clusters were generated by River, and if so retrieve the representative docs for the new clusters. BERtopic does not take this action after partial_fit, so I am running it manually. However, running the save method the second time results in an exception:

        if ensure_min_samples > 0:
            n_samples = _num_samples(array)
            if n_samples < ensure_min_samples:
>               raise ValueError(
                    "Found array with %d sample(s) (shape=%s) while a"
                    " minimum of %d is required%s."
                    % (n_samples, array.shape, ensure_min_samples, context)
                )
E               ValueError: Found array with 0 sample(s) (shape=(0, 1018)) while a minimum of 1 is required by the normalize function.

.venv/lib/python3.11/site-packages/sklearn/utils/validation.py:967: ValueError

This looks to be caused because there are zero samples in a cluster that was generated in the first partial_fit call in the second set of docs. In this case it would be great if the library skipped the empty cluster, and just emitted the representative docs it does have.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the _save_representative_docs entry point and reproduce the two partial_fit calls shown in the issue. Check how representative documents are selected when a cluster has zero samples; done means the second save completes and returns representative documents for non-empty clusters without the normalization exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.