LAION-AI / LAION-AI/Open-Assistant
Stat update slow for large database in the backend
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 37.4k
- Forks
- 3.3k
- PR merge metrics
- No merged PRs in 30d
Description
The current implementation of _update_stats_internal is quite slow in updating the stats.
As a proposal, we could let the database handle score updates and stat updates using a SQL Trigger on the text_label table. Additionally, we could also create an SQL function that can be executed by the backend to update the cached stats.
And if we implement a SQL trigger and function, we may need to reconsider how the cached_stats work

If the proposition seems interessting I could try to dev it 😄
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating _update_stats_internal and tracing how the text_label table and cached_stats are updated. Review the proposed SQL trigger and backend-invoked SQL function approach, then measure stat-update behavior on a large database. Done should include a concrete design and evidence that cached statistics remain correct while updates are faster.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sql
- Domain
- backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100