MaartenGr / MaartenGr/BERTopic

how to get ctf-idf formula inputs for top-10 topic words?

Open
#1,768 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

Hello Maarten,
Thank you for creating and maintaining BERTopic, it is an incredibly useful tool in my current work!
I want to ask if it is possible to obtain input components for ctf-idf formula for each of the top 10 words returned per topic. That is I would like to have actual values of tf_{x,c} , f_{x}, and A for each of the top-10 topic words. Below for completeness is a motivation for why I need these inputs in case you wondering or can offer an easier solution without tf_{x,c}.

I fit a topic model on a smaller representative news data and I want to use top-10 words in searches of similar articles in another very large news database to track each topics coverage intensity over time. It is infeasible to run the topic model on this larger data due to its huge size and because I do not have access to full text anyways. I can only submit text queries and count results over time ranges. Since not every document in a topic cluster contains each of the top-10 words I want to randomly draw smaller groups (eg tuples or triples) from top-10 words with probabilities proportional to their frequencies tf_{x,c} and then join the words within a group with an 'AND'. Then join several such groups by OR and submit this query for articles that contain all words from at least one random group.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named in the issue. Start by locating BERTopic's c-TF-IDF and topic-word representation entry points, then determine how tf_{x,c}, f_x, and A are computed for each topic's top 10 words. Done means those inputs are available for the requested words and their values can be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.