epam / epam/ai-dial-admin-evaluation-framework-backend
[Eval] Enriched test suite runs listing
- Dominant language
- Java
- Stars
- 3
- Forks
- 1
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 30
Description
# Enriched test suite run listing endpoint
**Depends on:** "Page-scoped total run cost from dial-adas" (the cost field only — everything else here
is independent and can be built in parallel).
## Context
`GET /api/v1/test-suite-runs` returns only meta-store run rows. The UI needs, for each run in one page:
the deployment it ran against, which metrics it produced, its overall score, and what it cost. Those
facts live in three places — `test_suite_runs.suite_snapshot` (meta, deliberately excluded from the list
projection to avoid TOAST decompression), `run_metric_snapshots` + `metric_score_result` (analytics), and
dial-adas usage logs — so a client needs 1 + 3N calls to assemble that view today.
This adds a **separate** endpoint that fans in server-side. The existing list endpoint and its hot path
stay untouched.
## Endpoint
`GET /api/v1/test-suite-runs/enriched` → `PageResponseDto`.
Routing: under Boot 4's `PathPatternParser` a literal segment outranks `{id}`, so this never reaches
`getRun(@PathVariable UUID id)` — same precedent as `/api/v1/deployments/all/**`. Pin it with a test.
### Pagination (explicitly in scope)
Identical offset contract to the plain list, reusing `PaginationParamResolver`,
`FilterWhitelists.TEST_SUITE_RUNS`, `SortWhitelists.TEST_SUITE_RUNS`,
`@FilterParam(max = MAX_FILTER_PARAMS)` and `@Size(max = MAX_SORT_PARAMS)`, plus three endpoint-specific
rules:
1. **Enrichment is scoped to the page's run ids.** The page is read first; every downstream read takes
exactly that id set.
2. **Enriched fields are not filterable or sortable** — they live in other stores, so sorting on them
would mean reading every run. State it in the OpenAPI description; do not extend the whitelists.
3. **Its own, smaller page cap**: `test-suite-run.enriched-list.default-page-size: 25`,
`max-page-size: 50` (validated `<= pagination.max-size`; over-cap `size` → 400 `VALIDATION_ERROR`).
Two reasons: each page detoasts `size` `suite_snapshot` values server-side (Postgres cannot partially
detoast a `jsonb` datum — SQL-side key extraction saves transfer and parsing, not decompression), and
the adas filter carries one `co` predicate per run id, so a large page becomes a wide `OR` over a
usage-log scan.
Add the overload to the shared resolver, never in the controller:
`PaginationParamResolver.resolveSize(Integer size, int defaultSize, int maxSize)`.
`includeTotalCount` behaves exactly as today (count query on the meta side only).
### Response item
`service/domain/dto/EnrichedTestSuiteRunResponseDto` — `@JsonUnwrapped TestSuiteRunResponseDto run` plus
the enrichment fields, so the wire shape is a strict superset of the plain list and no base field is
duplicated:
```json
{
"id": "...", "testSuiteId": "...", "status": "COMPLETED", "...": "(all existing list fields)",
"suiteType": "DEPLOYMENT",
"deploymentRef": {"id": "deploy-001", "name": "Production Deployment", "version": "1.0", "type": "dial-application"},
"mcpDeploymentRef": null,
"metricNames": ["Accuracy", "Relevance"],
"overallScore": 0.82,
"totalCost": 1.4823,
"costStatus": "AVAILABLE"
}
```
Two traps:
- The app's `JsonMapper` is globally `NON_NULL`, so a degraded `"totalCost": null` would be **silently
dropped** — exactly the state the client must see. Annotate `totalCost`, `costStatus` and
`overallScore` `@JsonInclude(ALWAYS)` and cover it with a serialization unit test.
- The codebase has no existing `@JsonUnwrapped` usage (`AggregatedMetricDefinitionResponseDto` is the
enriched-DTO precedent and duplicates its base fields), so assert in the OpenAPI test that springdoc
flattens it. If it does not render cleanly, fall back to a nested `run` object — a one-line DTO change
with no service or repository impact.
`metricNames` defaults to `[]` for a metric-less run, never null. `suiteSnapshot` stays null on this path
and is dropped by `NON_NULL`, so no snapshot payload leaks into the response.
## Scope
### 1. Meta — deployment refs in the same query as the page (`data.db`)
Do **not** add a second by-ids round trip: the detoast happens either way and two reads add a skew
window. Widen the projection in a *separate* method so the plain list's SQL is untouched.
- `data/db/repository/sql/SuiteSnapshotRefFields` — `jsonbGetAttributeAsText(SUITE_SNAPSHOT, "suiteType")`,
`jsonbGetAttribute(SUITE_SNAPSHOT, "deploymentRef")`, `jsonbGetAttribute(SUITE_SNAPSHOT, "mcpDeploymentRef")`,
reusing `PostgresJsonPathAccessor` (`data/db/repository/sql/json/`) rather than raw SQL.
- `PostgresTestSuiteRunRepository`: extract the body of `findAll` into a private
`findAllProjected(Field[] projection, RecordMapper<…,T> mapper, PageRequest, filters, includeTotalCount)`
so filter/sort/limit/offset/count semantics cannot drift, then add a third tier
`SELECT_LIST_WITH_SNAPSHOT_REFS_FIELDS = SELECT_LIST_FIELDS + SuiteSnapshotRefFields.ALL` and
`findAllWithSnapshotRefs(...)`. `findAll` must emit byte-identical SQL after the refactor.
- New model `record TestSuiteRunWithSnapshotRefs(TestSuiteRun run, String suiteType, String deploymentRefJson, String mcpDeploymentRefJson)`
— the data layer carries raw JSON, never DTOs. `TestSuiteRunRecordMapper.mapWithSnapshotRefs`.
- `TestSuiteRunMapper` deserializes the two fragments with the existing Jackson 3
`tools.jackson.databind.ObjectMapper`, mirroring its `deserializeSuiteSnapshot`. A PENDING run with
`suite_snapshot IS NULL` yields three nulls — do not throw.
- `TestSuiteRunService.listRunsWithSnapshotRefs(...)`, `@Transactional("metaTransactionManager", readOnly = true)`.
- Update `docs/patterns/selective-column-projection.md`: three tiers now, and note that extraction still
detoasts (it saves transfer and parsing).
### 2. Analytics — two bulk reads, each behind its own domain service
Cross-domain rule: the orchestrator calls analytics **services**, never their repositories. Each method
carries `@Transactional(value = "analyticsTransactionManager", readOnly = true)`.
- **Metric names, latest computation, one statement** —
`RunMetricSnapshotRepository.findLatestMetricNamesByRunIds(Collection)` → `Map>`:
a `DISTINCT ON (test_suite_run_id)` derived table picking `computation_id` by
`ORDER BY test_suite_run_id, computed_at_ms DESC, computation_id DESC`, joined back on
`(run, computation)`, selecting `tsmd_name`, `fetchGroups`. The `computation_id` tiebreak matters:
every row of one computation shares `computed_at_ms`, so two computations landing in the same
millisecond would otherwise resolve nondeterministically. jOOQ idiom to copy:
`PostgresMetricDeclarationVersionRepository.java:128-165`.
Resolving "latest" from the snapshot table is correct here — the metric-catalog exception blessed by
`docs/patterns/computation-versioning.md` (same as `EvalSummariesSchemaProvider`). Do **not** route it
through `ComputationResolver`.
- **Overall score** — `MetricScoreResultRepository.findLatestOverallByRunIds(Collection)` →
`Map`: `DISTINCT ON (test_suite_run_id) value` where
`metric_score_name = metric_name = 'overall'` (`constants/MetricScoreConstants`), same ordering and
tiebreak. An absent row → no entry → `overallScore: null`; that is the legitimate state for a run with
no `overallScore` definition and ≠1 numeric metric field, and it is not `0`.
- The two queries resolve "latest" independently (different tables, different timestamps). They agree
except across a concurrent recomputation — document that rather than coupling them, because binding the
score to the snapshot-derived computation makes a metric-less run's score unreadable.
- `MetricScoreService` gains its first read method; its class javadoc and
`openspec/specs/metric-score-statistics/spec.md` both currently say results are read **only** through
the Query DSL. Amend both.
- New migration `db/migration/analytics/POSTGRES/V1.__AddEnrichedRunListingIndexes.sql`:
```sql
CREATE INDEX idx_run_metric_snapshots_run_computed_at
ON run_metric_snapshots (test_suite_run_id, computed_at_ms DESC, computation_id);
CREATE INDEX idx_metric_score_result_overall_latest
ON metric_score_result (test_suite_run_id, computed_at_ms DESC, computation_id)
WHERE metric_score_name = 'overall' AND metric_name = 'overall';
```
Mirrors `idx_eval_summaries_run_computed_at` (V1.15). `idx_run_metric_snapshots_run` becomes a strict
prefix of the new index — dropping it is safe and saves write cost; raise it in review rather than
bundling it silently. Then `./gradlew generateJooq` and commit the generated diff (`Indexes.java` is
generated), update `docs/database-schema.md` (index rows + migration history), and consider adding
`METRIC_SCORE_RESULT` to `JooqSchemaDriftTest`'s analytics table list — it is missing today.
### 3. Orchestration (`service.domain`)
`EnrichedTestSuiteRunListService` — no class-level `@Transactional`; each step's transaction belongs to
the service it calls, mirroring `EvalSummaryExportService`'s documented "short meta tx, then short
analytics tx, then out-of-tx work" strategy. It injects **services only**, so the cross-domain rule holds
across datasources by construction.
```
1. meta tx (read-only) → page + snapshot refs, one statement (+ count if requested) COMMIT
2. empty page? → return immediately; no analytics query, no HTTP
3. analytics tx (read-only) → metric names for the page's ids COMMIT
4. analytics tx (read-only) → overall scores for the page's ids COMMIT
5. no transaction → RunCostFetcher.fetchTotalCosts(ids) ← the single HTTP call
6. assemble → EnrichedTestSuiteRunMapper
```
No DB connection is ever held across the HTTP call. If a single analytics snapshot across both reads is
later wanted, the only compliant way is one analytics-service method owning both queries — not a
`TransactionTemplate` wrapped around two service calls from the orchestrator.
### 4. Web / OpenAPI / config
- `TestSuiteRunController.listEnrichedRuns(...)` — same parameter binding as `listRuns` plus the
endpoint-specific size resolution. `@LogExecution` is already at class level.
- `configuration/OpenApiQueryParamCustomizer` — registry entry for `/api/v1/test-suite-runs/enriched`
with the same filter/sort specs and `PaginationType.OFFSET`, noting that enriched fields are neither
filterable nor sortable and that the page cap is endpoint-specific.
- Examples `api-v1-test-suite-runs-enriched-GET-response-200-minimal.json` and `-full.json`
(`minimal`/`full` are already allowed names in `OpenApiExampleCustomizer`). The `full` example should
show one MCP item and one degraded item (`totalCost: null`, `costStatus: "UNAVAILABLE"`).
- `application.yml` (defaults only here; `TestSuiteRunProperties.EnrichedList` holds structure and
validation): `default-page-size: 25`, `max-page-size: 50`. Rows in `docs/configuration.md` with all six
columns.
## Out of scope
- The dial-adas query itself and `RunCostFetcher` — see the dependency issue. If that issue concludes
adas cannot aggregate per run, this endpoint **ships without `totalCost`/`costStatus`**; every other
field is unaffected.
- Making enriched fields filterable or sortable.
- Any change to `GET /api/v1/test-suite-runs` or `GET /api/v1/test-suite-runs/{id}`.
## Acceptance criteria
- [ ] A page of N runs costs **3 SQL statements** (4 with `includeTotalCount`) and **at most 1** outbound
HTTP call, regardless of N.
- [ ] Metric names come from the latest computation only; a run with two computations shows only the
newer set.
- [ ] A run with no `overall` row returns `overallScore: null` with metrics still populated.
- [ ] MCP suites populate `mcpDeploymentRef` and leave `deploymentRef` null, keyed by `suiteType`.
- [ ] adas unavailable → HTTP **200** with `totalCost: null`, `costStatus: "UNAVAILABLE"`, every other
field intact.
- [ ] `size` above the cap → 400; omitted `size` → the enriched default, not the global 100.
- [ ] `GET /api/v1/test-suite-runs/{id}` still resolves, and the plain list's SQL is unchanged.
- [ ] Empty page → 200 with no analytics query and no adas call.
## Tests
Unit — `EnrichedTestSuiteRunListServiceTest` (empty page short-circuits, `verifyNoInteractions` on both
analytics services and the cost fetcher; a run missing from every enrichment map still appears with nulls
and `metricNames: []`); mapper tests (DEPLOYMENT vs MCP_TOOL snapshot; null snapshot → three nulls);
serialization test against the app's `JsonMapper` asserting `"totalCost": null`, `"overallScore": null`,
`"costStatus"` are all emitted, run fields are top-level, and `suiteSnapshot` is absent.
Functional — `functional/tests/EnrichedTestSuiteRunListFunctionalTests`, registered as a `@Nested` class
in `PostgresFunctionalTests`, with `DialAdasClient` stubbed via the existing `@MockitoBean`:
happy path; two computations → newer only; no `overall` row; MCP suite; adas throws → 200 + degraded
cost; **exactly one adas call per page** (`verify(..., times(1))` — the requirement most likely to
regress silently); paging/filter/sort parity with the plain list incl. `includeTotalCount`; page-cap 400;
empty page → no adas call; routing guard; `/v3/api-docs` carries the endpoint with both examples.
`PostgresTestSuiteRunRepositoryFunctionalTests` gains a case that `findAllWithSnapshotRefs` returns the
refs while `findAll` still returns `suiteSnapshot == null`.
`MetaTestDataHelper.createTestSuiteRun` writes a snapshot **without** `deploymentRef`; add an
intent-named helper overload that seeds the refs. No raw SQL or snapshot JSON inside test methods.
Contributor guide
Assessment
This issue has not been assessed yet.