grpc: WatchJobs, a server stream instead of a poll loop
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 20
- Forks
- 2
- Avg merge
- 5h 10m
- Merged PRs (30d)
- 127
Description
An embedded SDK awaits a result in-process. A remote client cannot: the only way to learn that a
job finished is to call GET /v1/jobs/{id} in a loop.
That is wrong on three counts. It costs a database read per poll per client. It bounds latency
by the poll interval rather than by the work. And it makes the obvious client-side helper —
"enqueue this and give me the answer" — a thing every remote SDK has to reinvent, badly and
differently.
Add a server-streaming WatchJobs to the producer door:
- watch by job id, by a set of ids, or by queue
- emit on state transitions, terminal or not, so progress is visible and not just completion
- resumable: a client that reconnects must be able to say where it left off and not miss a
terminal event in the gap - bounded: a stream is a resource. Server-side limits on concurrent watches per token, and a
documented behaviour when a client stops reading
Two things to settle in the design:
- How the stream learns of a change. The scheduler already produces
ResultOutcome, and
there is a pub/sub mechanism in the core. Reuse one of them rather than adding a third; a
polling loop hidden behind a stream is the same cost with a nicer API. - Terminal semantics. A watch on a job that is already terminal must emit and close, not
hang. A watch on an id that does not exist, or is in another namespace, must be
indistinguishable — the id-addressed reads already take this position.
This unblocks flexiq tail in #832 and removes the poll loop from every remote client.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the producer door, the existing GET job-status path, ResultOutcome, and the core pub/sub mechanism. Compare the available change-notification paths and settle the stream's filtering, resume, terminal, concurrency, and stopped-reader behavior. Done means WatchJobs is documented with those semantics and no polling loop is used.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- api, backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100