randomparity / randomparity/kdive

Account object-store storage cost at project/investigation/user granularity

Open
#1,504 1 comment 0 reactions 0 assignees View on GitHub
area:allocation area:core-platform effort:L priority:P2 risk:daytime-only status:ready type:feature
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

## Problem / motivation

Agent-uploaded artifacts — custom rootfs images (#743, multi-GiB), build artifacts, vmcores, pcaps — impose a **real, unbounded object-storage cost** that is currently **not accounted or attributed**. The Allocation plane tracks compute (vcpu/mem/disk quotas, billing intervals), but there is no equivalent for **bytes-at-rest in the object store** per project / investigation / user.

Consequences today:

- A project can upload arbitrarily many large blobs with no per-owner storage quota or visibility.
- Leaked/orphaned objects (e.g. #1501) accrue cost silently — nothing surfaces "this project is holding N GiB of SENSITIVE rootfs images".
- No signal to drive retention/reclaim decisions (which investigations dominate storage, which users, which artifact classes).

## Proposal (needs design + ADR)

Track object-store bytes-at-rest as a first-class accounted resource, attributed at **project / investigation / user** granularity:

- **Measure.** Sum stored object sizes by `owner_kind`/`owner_id`, rolled up to project, investigation, and uploading user. Source of truth: the `artifacts` rows (they carry the object key + owner) plus size (record `size_bytes` at commit if not already persisted per object).
- **Expose.** A read surface (MCP tool + metrics) reporting current storage per project/investigation/user and per artifact class (rootfs / build / vmcore / pcap). Feeds dashboards and retention decisions.
- **Optionally enforce.** Per-project (and/or per-investigation) storage budgets in the Allocation plane, admission-checked at `create_*_upload` time, mirroring how compute quotas gate allocation. Start with measure+expose; gate later.

## Design questions

- Reuse the Allocation/accounting plane (quotas/budgets) vs. a separate storage-accounting service.
- Where size is authoritative (HEAD the store vs. persisted `artifacts.size_bytes`) and how to stay correct under multipart/composite objects and best-effort-delete orphans.
- Whether enforcement blocks uploads (hard quota) or only warns/reports (soft), and the operator override path.
- Interaction with retention/gc (ADR-0234 investigation-artifact expiry) — accounting should reflect what reclaim will free.

## Acceptance (high level; refine in spec)

- Storage bytes-at-rest are queryable per project, per investigation, and per uploading user, broken down by artifact class.
- The numbers reconcile against the object store within a bounded tolerance.
- (If enforcement is in scope) an upload that would exceed a configured per-project/investigation storage budget is rejected at declaration with a clear message.

## Context

Motivated by #743 / PR #1499 introducing large (multi-GiB) agent-uploaded rootfs images. Related: #1501 (orphaned SENSITIVE blobs accrue untracked cost), the investigation-scoped-rootfs follow-up, ADR-0234 (investigation-artifact retention). Accounting/quota precedent lives in the Allocation plane (`area:allocation`).

Contributor guide

Open the contributing guide

Research direction

Start by reading the Allocation plane precedent, the artifacts data model, ADR-0234, and the create_*_upload entry points named in the issue. Define the accounting design and ADR first; done means storage is queryable by project, investigation, user, and artifact class, reconciles with the object store within a bounded tolerance, and any scoped enforcement has a clear rejection path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.