randomparity / randomparity/kdive

Per-principal rate limit for check_ssh_reachable probe jobs

Open
#982 0 comments 0 reactions 0 assignees View on GitHub
area:core-platform effort:M priority:P3 risk:daytime-only status:blocked type:feature
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

## Context

Follow-up from #972 / PR#981 (ADR-0298). `systems.check_ssh_reachable` is the first **VIEWER-gated** MCP tool that enqueues a durable worker job, and it uses a **fresh-nonce `dedup_key`** so each call mints a distinct job — this is deliberate: a liveness probe must be a fresh point-in-time measurement, never pinned to a prior job's stale verdict.

The accepted cost, documented in ADR-0298, is that probe jobs do **not** coalesce. A VIEWER that loops `check_ssh_reachable` can enqueue one bounded probe job per call into the shared worker queue (which also serves provision/build). Each job is bounded — a single connect with a 15s deadline and no retry beyond that — and the load is observable via `jobs.list` depth, but there is currently no per-caller cap.

## Ask

Add a per-principal (or per-project) rate limit / quota for `check_ssh_reachable` probe enqueues, so a looping low-privilege caller cannot crowd out provision/build work in the shared queue.

This is a **conditional** follow-up: only worth doing if observed `jobs.list` `check_ssh_reachable` depth shows real queue pressure in practice. Filing it so the residual is tracked rather than lost.

## Notes / design constraints

- Do **not** reintroduce coalescing dedup as the fix — freshness is the hard requirement for a liveness probe (ADR-0298 rejected in-flight coalescing as it would return stale verdicts).
- A rate limit is the right lever: it bounds enqueue rate without collapsing distinct measurements.
- Consider whether the limit belongs at the tool layer (reject the enqueue) or as a queue-admission concern for probe-kind jobs.

Refs #972, ADR-0298.

Contributor guide

Open the contributing guide

Research direction

Start by reading ADR-0298 and the behavior of systems.check_ssh_reachable, then inspect how jobs.list exposes probe depth in the shared worker queue. Decide whether admission belongs at the tool or queue layer, add a per-principal or per-project cap without restoring deduplication, and verify that fresh probe measurements remain distinct while provision/build work is protected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.