divviup / divviup/janus

Database query improvements for fixed-size task aggregate job creation

Open
#688 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

database
Dominant language
Rust
Stars
70
Forks
16
Avg merge
4h 36m
Merged PRs (30d)
30

Description

Continuing from discussion on #658: The method read_batch_size(), used during aggregate job creation for fixed-size tasks, may create a lot of SIReadLocks when doing index scans over report_aggregations. We discussed a couple denormalizing strategies that could be useful tradeoffs: either storing the number of report aggregations in different states in the aggregation_jobs table, or storing min_size/max_size in outstanding_batches. create_aggregation_jobs_for_fixed_size_task_no_param() could also be broken up into multiple smaller transactions, if we ensure consistency with some other means, like read-modify-write or leader elections.

This is the fixed-size task counterpart of #269. (though I think the asymptotic complexity of the queries here is not an issue, compared to those in #269)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with read_batch_size() and create_aggregation_jobs_for_fixed_size_task_no_param(), then read the schema and query code for report_aggregations, aggregation_jobs, and outstanding_batches. Review issues #658 and #269 to understand the prior discussion; the work is not complete until a chosen strategy improves fixed-size aggregate job creation while preserving consistency.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.