How to Predict New Data Topics with BERTopic Model Loaded from Hugging Face in Python? (Automated solution)

Nobody has claimed this yet.

I've developed a BERTopic model for analyzing mobile app reviews and have successfully pushed it to Hugging Face. I'm able to load the model in my Python script but am facing challenges in predicting topics for new incoming feedback.

My goal is to automatically assign a topic to new app feedback using the trained BERTopic model. Here's the workflow I've implemented:

I load the BERTopic model from Hugging Face.
I use the SentenceTransformer model to encode the new documents (app feedback).
I attempt to transform the new documents using the loaded BERTopic model to predict their topics.

However, I'm uncertain about the last part of my code, where I aim to merge the predicted topics with the original dataset containing app feedback. I want to end up with a dataset that includes the original feedback and the assigned topic names, ideally with automatic incrementation for new data.

Here's the code snippet I'm using:

```
from huggingface_hub import login
from bertopic import BERTopic
from sentence_transformers import SentenceTransformer
import pandas as pd

# Login to Hugging Face
access_token_read = "hf_xynbKhivF.........."
login(token=access_token_read)

# Load the BERTopic model
model = BERTopic.load("shantanudave/BERTopic_ArXiv")

# Sample new document

new_docs = ["There is an issue with payment on the checkout."]

# Generate embeddings for the new document
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embedding_model.encode(new_docs, show_progress_bar=True)

# Predict the topic for the new document
new_topics, new_probs = model.transform(new_docs, embeddings)

# Code for merging predicted topics with the original dataset (uncertain part)
...

```

Questions:

Is the approach I'm using to predict and assign topics to new feedback using BERTopic correct?
How can I effectively merge the predicted topics back into the original dataset, ensuring that each piece of feedback gets its corresponding topic and topic name?

```
#My manual approach : -

T = topic_model.get_document_info(docs)
T_df = T[['Document', 'Topic', 'Name', 'CustomName']]
# Remove duplicates from this new DataFrame
T_clean = T_df.drop_duplicates(subset=['Document'], keep='first')
T_clean

# Performing a left join where 'Document' in T_clean matches 'eng_content' in df , 'eng_content' is the review/feedback

merged_df = pd.merge(df, T_clean, how='left', left_on='eng_content', right_on='Document')
topic_doc_df_1 = merged_df
# Rename columns
topic_doc_df_1 = topic_doc_df_1.rename(columns={
'CustomName': 'Topic_name',
'score': 'Rating',
'date': 'Date',
'language': 'Language',
'sentiment': 'Sentiment',
'probability': 'Probability',
'app_version': 'App_version'
})

# Rearrange columns
topic_doc_df_1 = topic_doc_df_1[['Topic', 'Topic_name', 'Document', 'Rating', 'Date', 'Language', 'Sentiment', 'Probability', 'App_version']]

topic_doc_df_1

```
I have another function clean_doc() which basically reads new data from df['eng_content'] and provide list 'new_docs'

I am sure there must be a better way of doing this that I am missing,

Thanks in advance,
Shantanu

No repository file, test, or entry point is identified; the issue asks for guidance on loading a BERTopic model, transforming new feedback, and merging predictions into a pandas dataset. Start by reviewing the BERTopic transform and document-information APIs, and consider whether the requested workflow belongs in a documented example; done means the prediction and merge process is clearly explained.

Written by the indexing model from the issue text.