Add a stats table to eliminate need for parity-counting query
- Dominant language
- Jupyter Notebook
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Adding staging tables definitely helped speed up sentence parsing, but this has also revealed that the query counting how many sentences we have in each class is another huge bottleneck, possibly the original big one I was hoping to eliminate with the stg tables.
Let's actually add a function to handle that stats table. Stats table will include a last_updated_date for each metric. This way we can also add logic for updating the metrics at arbitrary intervals.
Contributor guide
No contributing guide indexed for this repository
Research direction
Inspect the staging-table code and the query that counts sentences in each class. Define how the stats table stores each metric and its last_updated_date, then verify that the metrics can be refreshed at arbitrary intervals without relying on the parity-counting query.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook
- Domain
- database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100