MaartenGr / MaartenGr/BERTopic
how to get ctf-idf formula inputs for top-10 topic words?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hello Maarten,
Thank you for creating and maintaining BERTopic, it is an incredibly useful tool in my current work!
I want to ask if it is possible to obtain input components for ctf-idf formula for each of the top 10 words returned per topic. That is I would like to have actual values of tf_{x,c} , f_{x}, and A for each of the top-10 topic words. Below for completeness is a motivation for why I need these inputs in case you wondering or can offer an easier solution without tf_{x,c}.
I fit a topic model on a smaller representative news data and I want to use top-10 words in searches of similar articles in another very large news database to track each topics coverage intensity over time. It is infeasible to run the topic model on this larger data due to its huge size and because I do not have access to full text anyways. I can only submit text queries and count results over time ranges. Since not every document in a topic cluster contains each of the top-10 words I want to randomly draw smaller groups (eg tuples or triples) from top-10 words with probabilities proportional to their frequencies tf_{x,c} and then join the words within a group with an 'AND'. Then join several such groups by OR and submit this query for articles that contain all words from at least one random group.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named in the issue. Start by locating BERTopic's c-TF-IDF and topic-word representation entry points, then determine how tf_{x,c}, f_x, and A are computed for each topic's top 10 words. Done means those inputs are available for the requested words and their values can be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100