Separate real-time and batch priority requests to analyze packages
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 912
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
I started a backfill of 95k packages from crates.io.
These enter as messages in the same PubSub queue as the live "real-time" messages for packages being updated.
Unfortunately this causes the "real-time" requests to be delayed, potentially missing detections for packages that are only live for a short time on a package repository.
Ideally there should be a way to priotize "batch" messages from "real-time" messages, so the real-time messages can be processed first (or at a higher rate).
Alternatively separate "batch" analysis workers could be spun-up when batch work is present.
Either way it would be good if batch work didn't block real-time work.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names a shared PubSub queue and analysis workers but no files or tests. Start by locating the queue-consumption and worker entry points, then determine how batch and real-time messages are identified. Done means batch analysis no longer blocks real-time work, through prioritization or separate workers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100