MaartenGr / MaartenGr/BERTopic

Running Topics Over Time with just a subset of topics and documents

Open
#1,599 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

I'd like to run topic_model.topics_over_time() but only on a specific subset of documents and topics. Sometimes when working with a large corpus with lots of topics, running it on all documents and topics becomes a bit unwieldy especially if I'm only interested in the temporal variation of specific topics. Currently, that doesn't seem possible since topics_over_time doesn't allow for a manual selection of topics, and runs on the full list (specifically it references "Topic": self.topics_). I think this is easily solvable by changing the function so that topics is an argument, as follows:

``` python
def topics_over_time(self,
docs: List[str],
timestamps: Union[List[str],
List[int]],
topics: List[int] = None,
nr_bins: int = None,
datetime_format: str = None,
evolution_tuning: bool = True,
global_tuning: bool = True) -> pd.DataFrame:
```

And then changing the documents object as follows:

```python
documents = pd.DataFrame({"Document": docs, "Topic": topics, "Timestamps": timestamps})
```

I've forked the repo (https://github.com/yousufabdelfatah/BERTopic) and tested this out already because I was implementing this locally but thought that I should raise this as an issue for feedback and possible collaboration. Does this make sense or am I missing a potential issue that this causes downstream?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at topic_model.topics_over_time(), especially the current construction of the documents DataFrame and its use of self.topics_. Trace how document and topic selections are consumed downstream. Done means callers can provide document and topic subsets without breaking the existing default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.