epam / epam/ai-dial-admin-evaluation-framework-backend
[Eval] [techdebt] Offload oversized request/response payloads from JSONB to object storage with reference indirection
- Dominant language
- Java
- Stars
- 3
- Forks
- 1
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 30
Description
### Name and Version
0.2.0
### What is the problem this feature will solve?
Per-test-case `request_body` and `response_body` are stored inline as JSONB columns in the analytics database. With multimodal evaluations (image, audio, long-context document inputs and outputs), individual payloads can reach multiple megabytes — and a single suite run can produce thousands of such rows. This causes:
1. **Database bloat and TOAST overhead** — large JSONB rows force PostgreSQL into out-of-line TOAST storage, inflating storage cost and slowing list/aggregate queries that don't even need the body columns.
2. **Lossy mitigation today** — the runner truncates oversized responses at a configured byte cap (currently 5 MB) and persists a truncated payload. The original content is lost, which breaks reproducibility, debugging, and any downstream re-extraction of metrics or columns.
3. **Backup, replication, and migration pressure** — every dump/restore and every replica round-trip carries the full payload weight, even though most analytics queries never read those columns.
4. **No clean upper bound** — there is no hard cap on payload size at write time, so a single pathological run can disproportionately impact storage and query performance for unrelated runs.
### What is the feature you are proposing to solve the problem?
Introduce a **size-based offload path** for large request and response payloads: payloads above a configurable threshold (e.g. 256 KB) are written to object storage and the database row stores only a compact reference (URI + content type + size + hash) instead of the inline JSONB. Smaller payloads continue to be stored inline as today. The change applies symmetrically to **both `request_body` and `response_body`**, since both can carry multimodal content.
High-level behavior:
- A configurable size threshold determines whether a payload is stored inline or offloaded. Below the threshold, current inline JSONB behavior is preserved (no migration churn for typical text-only runs).
- Above the threshold, the payload is uploaded to object storage at a deterministic, run-scoped path, and the DB column stores a reference object describing where to fetch it, its media type, declared size, and a content hash for integrity.
- Read paths (detail views, exports, re-extraction, debugging endpoints) transparently resolve the reference on demand, so existing API consumers see the same payload content regardless of where it lives.
- A separate, larger hard cap replaces the current truncation-only model: payloads above the offload threshold but within the hard cap are stored losslessly in object storage; payloads exceeding the hard cap are rejected or recorded as an error rather than silently truncated.
- Lifecycle is tied to the parent run: deleting a run cascades to its offloaded payload objects so storage doesn't leak.
- A feature flag gates the new path and lets environments roll forward gradually; while disabled, behavior is unchanged.
- No breaking REST API changes — clients continue to receive payload content in the same shape; the indirection is internal.
### What alternatives have you considered?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.