epam / epam/ai-dial-admin-evaluation-framework-backend

[Eval] [techdebt] Offload oversized request/response payloads from JSONB to object storage with reference indirection

Open
#31 0 comments 0 reactions 1 assignee Claimed by @buhaiovos View on GitHub
enhancement
Dominant language
Java
Stars
3
Forks
1
Avg merge
1d 1h
Merged PRs (30d)
30

Description

### Name and Version

0.2.0

### What is the problem this feature will solve?

Per-test-case `request_body` and `response_body` are stored inline as JSONB columns in the analytics database. With multimodal evaluations (image, audio, long-context document inputs and outputs), individual payloads can reach multiple megabytes — and a single suite run can produce thousands of such rows. This causes:

1. **Database bloat and TOAST overhead** — large JSONB rows force PostgreSQL into out-of-line TOAST storage, inflating storage cost and slowing list/aggregate queries that don't even need the body columns.
2. **Lossy mitigation today** — the runner truncates oversized responses at a configured byte cap (currently 5 MB) and persists a truncated payload. The original content is lost, which breaks reproducibility, debugging, and any downstream re-extraction of metrics or columns.
3. **Backup, replication, and migration pressure** — every dump/restore and every replica round-trip carries the full payload weight, even though most analytics queries never read those columns.
4. **No clean upper bound** — there is no hard cap on payload size at write time, so a single pathological run can disproportionately impact storage and query performance for unrelated runs.

### What is the feature you are proposing to solve the problem?

Introduce a **size-based offload path** for large request and response payloads: payloads above a configurable threshold (e.g. 256 KB) are written to object storage and the database row stores only a compact reference (URI + content type + size + hash) instead of the inline JSONB. Smaller payloads continue to be stored inline as today. The change applies symmetrically to **both `request_body` and `response_body`**, since both can carry multimodal content.

High-level behavior:

- A configurable size threshold determines whether a payload is stored inline or offloaded. Below the threshold, current inline JSONB behavior is preserved (no migration churn for typical text-only runs).
- Above the threshold, the payload is uploaded to object storage at a deterministic, run-scoped path, and the DB column stores a reference object describing where to fetch it, its media type, declared size, and a content hash for integrity.
- Read paths (detail views, exports, re-extraction, debugging endpoints) transparently resolve the reference on demand, so existing API consumers see the same payload content regardless of where it lives.
- A separate, larger hard cap replaces the current truncation-only model: payloads above the offload threshold but within the hard cap are stored losslessly in object storage; payloads exceeding the hard cap are rejected or recorded as an error rather than silently truncated.
- Lifecycle is tied to the parent run: deleting a run cascades to its offloaded payload objects so storage doesn't leak.
- A feature flag gates the new path and lets environments roll forward gradually; while disabled, behavior is unchanged.
- No breaking REST API changes — clients continue to receive payload content in the same shape; the indirection is internal.

### What alternatives have you considered?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.