dmarx / dmarx/SuicideWatch

Add a stats table to eliminate need for parity-counting query

Open
#2 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Adding staging tables definitely helped speed up sentence parsing, but this has also revealed that the query counting how many sentences we have in each class is another huge bottleneck, possibly the original big one I was hoping to eliminate with the stg tables.

Let's actually add a function to handle that stats table. Stats table will include a last_updated_date for each metric. This way we can also add logic for updating the metrics at arbitrary intervals.

Contributor guide

No contributing guide indexed for this repository

Research direction

Inspect the staging-table code and the query that counts sentences in each class. Define how the stats table stores each metric and its last_updated_date, then verify that the metrics can be refreshed at arbitrary intervals without relying on the parity-counting query.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.