randomparity / randomparity/kdive
Account object-store storage cost at project/investigation/user granularity
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem / motivation
Agent-uploaded artifacts — custom rootfs images (#743, multi-GiB), build artifacts, vmcores, pcaps — impose a **real, unbounded object-storage cost** that is currently **not accounted or attributed**. The Allocation plane tracks compute (vcpu/mem/disk quotas, billing intervals), but there is no equivalent for **bytes-at-rest in the object store** per project / investigation / user.
Consequences today:
- A project can upload arbitrarily many large blobs with no per-owner storage quota or visibility.
- Leaked/orphaned objects (e.g. #1501) accrue cost silently — nothing surfaces "this project is holding N GiB of SENSITIVE rootfs images".
- No signal to drive retention/reclaim decisions (which investigations dominate storage, which users, which artifact classes).
## Proposal (needs design + ADR)
Track object-store bytes-at-rest as a first-class accounted resource, attributed at **project / investigation / user** granularity:
- **Measure.** Sum stored object sizes by `owner_kind`/`owner_id`, rolled up to project, investigation, and uploading user. Source of truth: the `artifacts` rows (they carry the object key + owner) plus size (record `size_bytes` at commit if not already persisted per object).
- **Expose.** A read surface (MCP tool + metrics) reporting current storage per project/investigation/user and per artifact class (rootfs / build / vmcore / pcap). Feeds dashboards and retention decisions.
- **Optionally enforce.** Per-project (and/or per-investigation) storage budgets in the Allocation plane, admission-checked at `create_*_upload` time, mirroring how compute quotas gate allocation. Start with measure+expose; gate later.
## Design questions
- Reuse the Allocation/accounting plane (quotas/budgets) vs. a separate storage-accounting service.
- Where size is authoritative (HEAD the store vs. persisted `artifacts.size_bytes`) and how to stay correct under multipart/composite objects and best-effort-delete orphans.
- Whether enforcement blocks uploads (hard quota) or only warns/reports (soft), and the operator override path.
- Interaction with retention/gc (ADR-0234 investigation-artifact expiry) — accounting should reflect what reclaim will free.
## Acceptance (high level; refine in spec)
- Storage bytes-at-rest are queryable per project, per investigation, and per uploading user, broken down by artifact class.
- The numbers reconcile against the object store within a bounded tolerance.
- (If enforcement is in scope) an upload that would exceed a configured per-project/investigation storage budget is rejected at declaration with a clear message.
## Context
Motivated by #743 / PR #1499 introducing large (multi-GiB) agent-uploaded rootfs images. Related: #1501 (orphaned SENSITIVE blobs accrue untracked cost), the investigation-scoped-rootfs follow-up, ADR-0234 (investigation-artifact retention). Accounting/quota precedent lives in the Allocation plane (`area:allocation`).
Contributor guide
Research direction
Start by reading the Allocation plane precedent, the artifacts data model, ADR-0234, and the create_*_upload entry points named in the issue. Define the accounting design and ADR first; done means storage is queryable by project, investigation, user, and artifact class, reconciles with the object store within a bounded tolerance, and any scoped enforcement has a clear rejection path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100