MaartenGr / MaartenGr/BERTopic
Allow recalculating representative docs with a configurable count (currently hardcoded to 3)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 921
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Feature request
_save_representative_docs hardcodes nr_samples=500 and nr_repr_docs=3. Users cannot control how many representative documents are stored per topic, nor the sampling pool size.
Add a public method to recalculate representative documents after training with a configurable count, following the API direction suggested in #1767:
# Recalculate with more representative docs after training
topic_model.recalculate_representative_docs(nr_repr_docs=10, nr_samples=1000)
Motivation
This is a frequently requested feature:
- #552 — "get more than 3 representative docs"
- #848 — "Ability to change # of documents pulled per topic"
- #1767 — "How to get more than 3 representative docs per topic"
In #1767, @MaartenGr noted:
"It might be nice to have an additional function that re-calculates the representative documents since this question seems to appear frequently."
PR #2130 attempted an nr_repr_docs constructor parameter, but @MaartenGr's feedback was that he prefers a standalone public function that recalculates representative docs after training.
Use cases: production users who need more docs for human review, small datasets where 500 exceeds topic size, users who want to iterate on representative doc quality post-training.
Your contribution
I can submit a PR following @MaartenGr's preferred API direction:
- New public method:
recalculate_representative_docs(nr_repr_docs=3, nr_samples=500)— recalculates and replacesrepresentative_docs_after training. - Constructor defaults
nr_repr_docs(default 3) andnr_repr_docs_nr_samples(default 500) — used duringfit_transform().
I've already been prototyping this in my fork along the lines you suggested, so I can open a PR quickly if this direction works for you.
Backward compatible: defaults reproduce the current behavior exactly.
Note: Works standalone. Pairs well with the sampling/indexing fix for _extract_representative_docs (#2495) — that fix corrects the sampling this method configures.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading _save_representative_docs, _extract_representative_docs, and fit_transform, since these are the entry points named in the issue. Implement the public recalculation path and confirm that representative_docs_ is replaced with the requested counts while default behavior remains unchanged; review PR #2130 for prior API context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100