MaartenGr / MaartenGr/BERTopic
Running Topics Over Time with just a subset of topics and documents
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
I'd like to run topic_model.topics_over_time() but only on a specific subset of documents and topics. Sometimes when working with a large corpus with lots of topics, running it on all documents and topics becomes a bit unwieldy especially if I'm only interested in the temporal variation of specific topics. Currently, that doesn't seem possible since topics_over_time doesn't allow for a manual selection of topics, and runs on the full list (specifically it references "Topic": self.topics_). I think this is easily solvable by changing the function so that topics is an argument, as follows:
``` python
def topics_over_time(self,
docs: List[str],
timestamps: Union[List[str],
List[int]],
topics: List[int] = None,
nr_bins: int = None,
datetime_format: str = None,
evolution_tuning: bool = True,
global_tuning: bool = True) -> pd.DataFrame:
```
And then changing the documents object as follows:
```python
documents = pd.DataFrame({"Document": docs, "Topic": topics, "Timestamps": timestamps})
```
I've forked the repo (https://github.com/yousufabdelfatah/BERTopic) and tested this out already because I was implementing this locally but thought that I should raise this as an issue for feedback and possible collaboration. Does this make sense or am I missing a potential issue that this causes downstream?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at topic_model.topics_over_time(), especially the current construction of the documents DataFrame and its use of self.topics_. Trace how document and topic selections are consumed downstream. Done means callers can provide document and topic subsets without breaking the existing default behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100