MaartenGr / MaartenGr/BERTopic
probabilities_ outcome not consistent with get_document_info output
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
Hi Maarten,
I found that `probabilities_` outcome is not always the same with the outcome of `get_document_info`.
`topic_model.probabilities_[0]` gave me the following:
`array([0.31045925, 0.0031012 , 0.00162018, 0.00349846, 0.0039572 ,
0.01974048, 0.00788453, 0.00418387, 0.00252376, 0.00240863,
0.00308729, 0.00263459, 0.00250224, 0.00254411, 0.00254329,
0.00283324, 0.00229805, 0.00638228, 0.00281222, 0.0028788 ,
0.00298958, 0.0030663 , 0.00246619, 0.00387198, 0.00227671,
0.00254577, 0.00245718, 0.0026639 , 0.00243602, 0.00419228,
0.00243791, 0.00244527, 0.00384736, 0.00285853, 0.00214743,
0.00218918, 0.0029944 ])`
From this output it seems that the dominant topic would be Topic 0.
However, `get_document_info` gave me this:
Document 在中国 嫦娥五号 探月工程 顺利实施 之际 美国 在航天 领域 频频 动作 退出_开放天空条...
Topic 5
Name 5_太空_卫星_外空_外空_攻防对抗
Representation [太空, 卫星, 外空, 外空_攻防对抗, 外空_攻防对抗 准备, 外空安全, 发射, 航天...
Representative_Docs [进入 世纪 以来 国际社会 围绕 外层空间 争夺 日益 激烈 外空安全 形势 日趋_紧张 ...
Top_n_words 太空 - 卫星 - 外空 - 外空_攻防对抗 - 外空_攻防对抗 准备 - 外空安全 - 发...
Probability 0.310459
Representative_document False
Name: 0, dtype: object
This output shows the same highest `Probability` (which is 0.310459) but assigns the document to Topic 5 instead of Topic 0.
Any solutions to this issue? Thanks much for the help!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported mismatch by comparing topic_model.probabilities_[0] with the Topic and Probability columns from get_document_info. Start by tracing the probability and topic-assignment handling behind these two entry points; done means the outputs agree for the reported document or the discrepancy is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100