MaartenGr / MaartenGr/BERTopic
Extract duplicated document-aggregation and topic-naming patterns into shared helpers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
### Feature request
`_bertopic.py` contains several near-identical copy-paste patterns that have diverged slightly over time. Extracting them into shared helpers would reduce duplication and prevent future divergence.
Three patterns are affected:
1. **Document aggregation** — `documents.groupby(["Topic"], as_index=False).agg({"Document": " ".join})` appears in 4 methods: `_extract_topics`, `hierarchical_topics`, `update_topics`, and `partial_fit`. When a new column needs aggregating (e.g., images), every call site must be updated independently.
2. **Feature names extraction** — a 5-line sklearn version check (`get_feature_names` vs `get_feature_names_out`) is duplicated in `hierarchical_topics` and `_c_tf_idf`.
3. **Topic name from words** — `"_".join([x[0] for x in words][:N])` appears in 4 places across 2 methods (`topic_labels_` with `[:4]`, `hierarchical_topics` with `[:5]` in 3 places), with inconsistent slicing.
### Motivation
Reduce maintenance burden. Any change to the aggregation pattern (e.g., adding image support) currently requires updating 4 call sites in lock-step. The duplicated sklearn version check is a maintenance liability — when the old API is eventually dropped, two separate locations need updating. Inconsistent topic name slicing could produce subtly different results depending on the code path.
### Your contribution
I can submit a PR that extracts three private helpers:
- `_aggregate_documents(documents, columns=None)` — consolidates the groupby pattern
- `_get_feature_names(vectorizer)` — wraps the sklearn version check
- `_topic_name_from_words(words, n=5)` — consistent slicing
**Zero behavior change.** Each helper is a direct extraction of existing code. All existing tests pass without modification.
I've already been prototyping this in my fork, so I can turn it into a PR quickly if the direction looks good to you.
---
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in _bertopic.py with _extract_topics, hierarchical_topics, update_topics, partial_fit, and _c_tf_idf, comparing the duplicated aggregation, feature-name, and topic-name patterns. Extract the three proposed private helpers without behavior changes, then run the existing tests and confirm they still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scikit-learn
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100