feat: Add topic model versioning and visualization (dummy data prototype)
- Dominant language
- Python
- Stars
- 13
- Forks
- 17
- Avg merge
- 6h 59m
- Merged PRs (30d)
- 1
Description
> [!NOTE]
> Migrated from [augurlabs/augur#3207](https://github.com/augurlabs/augur/issues/3207)
> Originally opened by `@xiaoha-cloud` on 2025-07-02
---
**Is your feature request related to a problem? If so, please describe the problem:**
Currently, Augur's topic modeling functionality does not support model versioning, metadata tracking, or visualization of topic models. This leads to several issues:
- It is difficult to track which model version produced which results.
- There is no way to compare different model runs or configurations.
- Users cannot easily visualize or interpret the topics generated by the model.
- Model evaluation metrics (such as coherence and perplexity) are not stored or accessible.
**Potential solutions:**
- Add a new `TopicModelMeta` table to store metadata for each topic model run (including method, parameters, scores, file paths, etc.).
- Add a `model_id` foreign key to the `TopicWord` and `RepoTopic` tables to associate words and topic distributions with a specific model version.
- Save model artifacts (pickle files, pyLDAvis HTML, wordcloud images) in a versioned directory structure under `artifacts/`.
- Generate and store topic visualizations (pyLDAvis, wordclouds) for each model.
- Store evaluation metrics (coherence, perplexity) in the database for each model.
- Provide helper functions to compare different models and retrieve their metadata.
**Additional context:**
- This feature is currently prototyped and tested on dummy data, not yet validated on production data.
- Main code changes are in:
- `augur/application/db/models/augur_data.py`
- `augur/application/db/models/__init__.py`
- `augur/tasks/data_analysis/clustering_worker/tasks.py`
- Example visualizations and model files are saved in the `artifacts/` directory.
- The goal is to make topic modeling results more reproducible, comparable, and interpretable for users and researchers.
Contributor guide
Assessment
This issue has not been assessed yet.