Hierarchical Clustering and Intertopic Distance Map

Nobody has claimed this yet.

Hello!
When I use the bertopic clustering, I find that the similar points in the intertopic distance map are not shown in the hierarchical clustering, which is that these two functions are very different, and ask for reasons.
Thank you!!

Attach the code:
import numpy as np
from bertopic import BERTopic
from transformers.pipelines import pipeline
from sentence_transformers import SentenceTransformer
from umap import UMAP
from hdbscan import HDBSCAN
from sklearn.feature_extraction.text import CountVectorizer

with open('./data/切词6.txt', 'r', encoding='utf-8') as file:
docs = file.readlines()
print('条数: ', len(docs))
print('预览第一条: ', docs[0])

vectorizer_model = None
embedding_model = embedding_model = model = SentenceTransformer("C:\\Users\\ASUS\\.cache\\huggingface\\hub\\models--sentence-transformers--all-MiniLM-L6-v2\\snapshots\\ea78891063587eb050ed4166b20062eaf978037c")
embeddings = np.load('c:\\Users\\ASUS\\Desktop\\BERTopic-Tutorial-main\\embedding\\emb6.npy')
print(embeddings.shape)

embeddings = embedding_model.encode(docs)
print('Embeddings shape:', embeddings.shape)

# 2. 创建UMAP降维模型
umap_model = UMAP(
n_neighbors=15,
n_components=5,
min_dist=0.0,
metric='cosine',
random_state=42 # ⚠️ 防止随机 https://maartengr.github.io/BERTopic/faq.html
)

# 3. 创建HDBSCAN聚类模型
hdbscan_model = HDBSCAN(
min_cluster_size=20,
min_samples=1,
metric='euclidean'
# predictio_data=True 算文档归属需要
)
# 定义停用词列表
stop_words = [
'Adolescent', 'adolescents', 'game', 'gaming', 'addiction', 'internet',
'study', 'research', 'analysis', 'findings', 'results', 'literature',
'review', 'impact', 'effects', 'factors', 'behavior', 'risk', 'prevention',
'treatment', 'gamer', 'games', 'children', 'disorder', 'gamers', 'child',
'student', 'students', 'play', 'player', 'plays', 'players', 'addictions',

]
# 初始化CountVectorizer并设置停用词
vectorizer_model = CountVectorizer(stop_words=stop_words)

# 6. 正式创建BERTopic模型
topic_model = BERTopic(
embedding_model=embedding_model,
vectorizer_model=vectorizer_model,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
nr_topics='auto'
)
# 查看主题
topics, probs = topic_model.fit_transform(docs, embeddings=embeddings) #传入训练好的词向量
topic_model.get_topic_info()
from bertopic.representation import MaximalMarginalRelevance # 导入
# diversity: How diverse the select keywords/keyphrases are.
# Values range between 0 and 1 with 0 being not diverse at all
# and 1 being most diverse.
representation_model = MaximalMarginalRelevance(diversity=0.5) # 创建mmr模型
topic_model = BERTopic(
embedding_model=embedding_model,
vectorizer_model=vectorizer_model,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
representation_model=representation_model # 传入模型
)

topics, probs = topic_model.fit_transform(docs, embeddings=embeddings)
topic_info = topic_model.get_topic_info()
topic_info
topic_docs = topic_model.get_document_info(docs)
topic_docs.to_csv('./聚类结果6.csv')
with open('./data/文本6.txt', 'r', encoding='utf-8') as file:
texts = file.readlines()
print('文本条数：', len(texts))
topic_docs.insert(1, '原文', texts)
with open('./data/时间6.txt', 'r', encoding='utf-8') as file:
years = file.readlines()
print('文本条数：', len(years))
topic_docs.insert(2, '时间', years)
topic_docs.to_csv('./聚类结果66.csv')
topic_model.visualize_barchart(top_n_topics=23, custom_labels=True)
topic_model.visualize_topics()
reduced_embeddings = UMAP(n_neighbors=10, n_components=2, min_dist=0.0, metric='cosine').fit_transform(embeddings)
topic_model.visualize_documents(docs, reduced_embeddings=reduced_embeddings, hide_document_hover=True)

hierarchical_topics = topic_model.hierarchical_topics(docs)
topic_model.visualize_hierarchy(hierarchical_topics=hierarchical_topics)

Start by reproducing the supplied script and compare the outputs of visualize_topics(), hierarchical_topics(docs), and visualize_hierarchy(hierarchical_topics). Read the behavior around these entry points and determine whether the differing relationships are expected or indicate a defect. Done means the reason is established and clearly documented, with a focused change identified if behavior is incorrect.

Written by the indexing model from the issue text.