epam / epam/ai-dial-admin-evaluation-framework-backend

[Eval] Enriched test suite runs listing

Open
#197 0 comments 0 reactions 1 assignee Claimed by @buhaiovos View on GitHub
enhancement P2 - High
Dominant language
Java
Stars
3
Forks
1
Avg merge
1d 1h
Merged PRs (30d)
30

Description

# Enriched test suite run listing endpoint

**Depends on:** "Page-scoped total run cost from dial-adas" (the cost field only — everything else here
is independent and can be built in parallel).

## Context

`GET /api/v1/test-suite-runs` returns only meta-store run rows. The UI needs, for each run in one page:
the deployment it ran against, which metrics it produced, its overall score, and what it cost. Those
facts live in three places — `test_suite_runs.suite_snapshot` (meta, deliberately excluded from the list
projection to avoid TOAST decompression), `run_metric_snapshots` + `metric_score_result` (analytics), and
dial-adas usage logs — so a client needs 1 + 3N calls to assemble that view today.

This adds a **separate** endpoint that fans in server-side. The existing list endpoint and its hot path
stay untouched.

## Endpoint

`GET /api/v1/test-suite-runs/enriched` → `PageResponseDto`.

Routing: under Boot 4's `PathPatternParser` a literal segment outranks `{id}`, so this never reaches
`getRun(@PathVariable UUID id)` — same precedent as `/api/v1/deployments/all/**`. Pin it with a test.

### Pagination (explicitly in scope)

Identical offset contract to the plain list, reusing `PaginationParamResolver`,
`FilterWhitelists.TEST_SUITE_RUNS`, `SortWhitelists.TEST_SUITE_RUNS`,
`@FilterParam(max = MAX_FILTER_PARAMS)` and `@Size(max = MAX_SORT_PARAMS)`, plus three endpoint-specific
rules:

1. **Enrichment is scoped to the page's run ids.** The page is read first; every downstream read takes
exactly that id set.
2. **Enriched fields are not filterable or sortable** — they live in other stores, so sorting on them
would mean reading every run. State it in the OpenAPI description; do not extend the whitelists.
3. **Its own, smaller page cap**: `test-suite-run.enriched-list.default-page-size: 25`,
`max-page-size: 50` (validated `<= pagination.max-size`; over-cap `size` → 400 `VALIDATION_ERROR`).
Two reasons: each page detoasts `size` `suite_snapshot` values server-side (Postgres cannot partially
detoast a `jsonb` datum — SQL-side key extraction saves transfer and parsing, not decompression), and
the adas filter carries one `co` predicate per run id, so a large page becomes a wide `OR` over a
usage-log scan.

Add the overload to the shared resolver, never in the controller:
`PaginationParamResolver.resolveSize(Integer size, int defaultSize, int maxSize)`.

`includeTotalCount` behaves exactly as today (count query on the meta side only).

### Response item

`service/domain/dto/EnrichedTestSuiteRunResponseDto` — `@JsonUnwrapped TestSuiteRunResponseDto run` plus
the enrichment fields, so the wire shape is a strict superset of the plain list and no base field is
duplicated:

```json
{
"id": "...", "testSuiteId": "...", "status": "COMPLETED", "...": "(all existing list fields)",
"suiteType": "DEPLOYMENT",
"deploymentRef": {"id": "deploy-001", "name": "Production Deployment", "version": "1.0", "type": "dial-application"},
"mcpDeploymentRef": null,
"metricNames": ["Accuracy", "Relevance"],
"overallScore": 0.82,
"totalCost": 1.4823,
"costStatus": "AVAILABLE"
}
```

Two traps:

- The app's `JsonMapper` is globally `NON_NULL`, so a degraded `"totalCost": null` would be **silently
dropped** — exactly the state the client must see. Annotate `totalCost`, `costStatus` and
`overallScore` `@JsonInclude(ALWAYS)` and cover it with a serialization unit test.
- The codebase has no existing `@JsonUnwrapped` usage (`AggregatedMetricDefinitionResponseDto` is the
enriched-DTO precedent and duplicates its base fields), so assert in the OpenAPI test that springdoc
flattens it. If it does not render cleanly, fall back to a nested `run` object — a one-line DTO change
with no service or repository impact.

`metricNames` defaults to `[]` for a metric-less run, never null. `suiteSnapshot` stays null on this path
and is dropped by `NON_NULL`, so no snapshot payload leaks into the response.

## Scope

### 1. Meta — deployment refs in the same query as the page (`data.db`)

Do **not** add a second by-ids round trip: the detoast happens either way and two reads add a skew
window. Widen the projection in a *separate* method so the plain list's SQL is untouched.

- `data/db/repository/sql/SuiteSnapshotRefFields` — `jsonbGetAttributeAsText(SUITE_SNAPSHOT, "suiteType")`,
`jsonbGetAttribute(SUITE_SNAPSHOT, "deploymentRef")`, `jsonbGetAttribute(SUITE_SNAPSHOT, "mcpDeploymentRef")`,
reusing `PostgresJsonPathAccessor` (`data/db/repository/sql/json/`) rather than raw SQL.
- `PostgresTestSuiteRunRepository`: extract the body of `findAll` into a private
`findAllProjected(Field[] projection, RecordMapper<…,T> mapper, PageRequest, filters, includeTotalCount)`
so filter/sort/limit/offset/count semantics cannot drift, then add a third tier
`SELECT_LIST_WITH_SNAPSHOT_REFS_FIELDS = SELECT_LIST_FIELDS + SuiteSnapshotRefFields.ALL` and
`findAllWithSnapshotRefs(...)`. `findAll` must emit byte-identical SQL after the refactor.
- New model `record TestSuiteRunWithSnapshotRefs(TestSuiteRun run, String suiteType, String deploymentRefJson, String mcpDeploymentRefJson)`
— the data layer carries raw JSON, never DTOs. `TestSuiteRunRecordMapper.mapWithSnapshotRefs`.
- `TestSuiteRunMapper` deserializes the two fragments with the existing Jackson 3
`tools.jackson.databind.ObjectMapper`, mirroring its `deserializeSuiteSnapshot`. A PENDING run with
`suite_snapshot IS NULL` yields three nulls — do not throw.
- `TestSuiteRunService.listRunsWithSnapshotRefs(...)`, `@Transactional("metaTransactionManager", readOnly = true)`.
- Update `docs/patterns/selective-column-projection.md`: three tiers now, and note that extraction still
detoasts (it saves transfer and parsing).

### 2. Analytics — two bulk reads, each behind its own domain service

Cross-domain rule: the orchestrator calls analytics **services**, never their repositories. Each method
carries `@Transactional(value = "analyticsTransactionManager", readOnly = true)`.

- **Metric names, latest computation, one statement** —
`RunMetricSnapshotRepository.findLatestMetricNamesByRunIds(Collection)` → `Map>`:
a `DISTINCT ON (test_suite_run_id)` derived table picking `computation_id` by
`ORDER BY test_suite_run_id, computed_at_ms DESC, computation_id DESC`, joined back on
`(run, computation)`, selecting `tsmd_name`, `fetchGroups`. The `computation_id` tiebreak matters:
every row of one computation shares `computed_at_ms`, so two computations landing in the same
millisecond would otherwise resolve nondeterministically. jOOQ idiom to copy:
`PostgresMetricDeclarationVersionRepository.java:128-165`.

Resolving "latest" from the snapshot table is correct here — the metric-catalog exception blessed by
`docs/patterns/computation-versioning.md` (same as `EvalSummariesSchemaProvider`). Do **not** route it
through `ComputationResolver`.
- **Overall score** — `MetricScoreResultRepository.findLatestOverallByRunIds(Collection)` →
`Map`: `DISTINCT ON (test_suite_run_id) value` where
`metric_score_name = metric_name = 'overall'` (`constants/MetricScoreConstants`), same ordering and
tiebreak. An absent row → no entry → `overallScore: null`; that is the legitimate state for a run with
no `overallScore` definition and ≠1 numeric metric field, and it is not `0`.
- The two queries resolve "latest" independently (different tables, different timestamps). They agree
except across a concurrent recomputation — document that rather than coupling them, because binding the
score to the snapshot-derived computation makes a metric-less run's score unreadable.
- `MetricScoreService` gains its first read method; its class javadoc and
`openspec/specs/metric-score-statistics/spec.md` both currently say results are read **only** through
the Query DSL. Amend both.
- New migration `db/migration/analytics/POSTGRES/V1.__AddEnrichedRunListingIndexes.sql`:

```sql
CREATE INDEX idx_run_metric_snapshots_run_computed_at
ON run_metric_snapshots (test_suite_run_id, computed_at_ms DESC, computation_id);
CREATE INDEX idx_metric_score_result_overall_latest
ON metric_score_result (test_suite_run_id, computed_at_ms DESC, computation_id)
WHERE metric_score_name = 'overall' AND metric_name = 'overall';
```

Mirrors `idx_eval_summaries_run_computed_at` (V1.15). `idx_run_metric_snapshots_run` becomes a strict
prefix of the new index — dropping it is safe and saves write cost; raise it in review rather than
bundling it silently. Then `./gradlew generateJooq` and commit the generated diff (`Indexes.java` is
generated), update `docs/database-schema.md` (index rows + migration history), and consider adding
`METRIC_SCORE_RESULT` to `JooqSchemaDriftTest`'s analytics table list — it is missing today.

### 3. Orchestration (`service.domain`)

`EnrichedTestSuiteRunListService` — no class-level `@Transactional`; each step's transaction belongs to
the service it calls, mirroring `EvalSummaryExportService`'s documented "short meta tx, then short
analytics tx, then out-of-tx work" strategy. It injects **services only**, so the cross-domain rule holds
across datasources by construction.

```
1. meta tx (read-only) → page + snapshot refs, one statement (+ count if requested) COMMIT
2. empty page? → return immediately; no analytics query, no HTTP
3. analytics tx (read-only) → metric names for the page's ids COMMIT
4. analytics tx (read-only) → overall scores for the page's ids COMMIT
5. no transaction → RunCostFetcher.fetchTotalCosts(ids) ← the single HTTP call
6. assemble → EnrichedTestSuiteRunMapper
```

No DB connection is ever held across the HTTP call. If a single analytics snapshot across both reads is
later wanted, the only compliant way is one analytics-service method owning both queries — not a
`TransactionTemplate` wrapped around two service calls from the orchestrator.

### 4. Web / OpenAPI / config

- `TestSuiteRunController.listEnrichedRuns(...)` — same parameter binding as `listRuns` plus the
endpoint-specific size resolution. `@LogExecution` is already at class level.
- `configuration/OpenApiQueryParamCustomizer` — registry entry for `/api/v1/test-suite-runs/enriched`
with the same filter/sort specs and `PaginationType.OFFSET`, noting that enriched fields are neither
filterable nor sortable and that the page cap is endpoint-specific.
- Examples `api-v1-test-suite-runs-enriched-GET-response-200-minimal.json` and `-full.json`
(`minimal`/`full` are already allowed names in `OpenApiExampleCustomizer`). The `full` example should
show one MCP item and one degraded item (`totalCost: null`, `costStatus: "UNAVAILABLE"`).
- `application.yml` (defaults only here; `TestSuiteRunProperties.EnrichedList` holds structure and
validation): `default-page-size: 25`, `max-page-size: 50`. Rows in `docs/configuration.md` with all six
columns.

## Out of scope

- The dial-adas query itself and `RunCostFetcher` — see the dependency issue. If that issue concludes
adas cannot aggregate per run, this endpoint **ships without `totalCost`/`costStatus`**; every other
field is unaffected.
- Making enriched fields filterable or sortable.
- Any change to `GET /api/v1/test-suite-runs` or `GET /api/v1/test-suite-runs/{id}`.

## Acceptance criteria

- [ ] A page of N runs costs **3 SQL statements** (4 with `includeTotalCount`) and **at most 1** outbound
HTTP call, regardless of N.
- [ ] Metric names come from the latest computation only; a run with two computations shows only the
newer set.
- [ ] A run with no `overall` row returns `overallScore: null` with metrics still populated.
- [ ] MCP suites populate `mcpDeploymentRef` and leave `deploymentRef` null, keyed by `suiteType`.
- [ ] adas unavailable → HTTP **200** with `totalCost: null`, `costStatus: "UNAVAILABLE"`, every other
field intact.
- [ ] `size` above the cap → 400; omitted `size` → the enriched default, not the global 100.
- [ ] `GET /api/v1/test-suite-runs/{id}` still resolves, and the plain list's SQL is unchanged.
- [ ] Empty page → 200 with no analytics query and no adas call.

## Tests

Unit — `EnrichedTestSuiteRunListServiceTest` (empty page short-circuits, `verifyNoInteractions` on both
analytics services and the cost fetcher; a run missing from every enrichment map still appears with nulls
and `metricNames: []`); mapper tests (DEPLOYMENT vs MCP_TOOL snapshot; null snapshot → three nulls);
serialization test against the app's `JsonMapper` asserting `"totalCost": null`, `"overallScore": null`,
`"costStatus"` are all emitted, run fields are top-level, and `suiteSnapshot` is absent.

Functional — `functional/tests/EnrichedTestSuiteRunListFunctionalTests`, registered as a `@Nested` class
in `PostgresFunctionalTests`, with `DialAdasClient` stubbed via the existing `@MockitoBean`:
happy path; two computations → newer only; no `overall` row; MCP suite; adas throws → 200 + degraded
cost; **exactly one adas call per page** (`verify(..., times(1))` — the requirement most likely to
regress silently); paging/filter/sort parity with the plain list incl. `includeTotalCount`; page-cap 400;
empty page → no adas call; routing guard; `/v3/api-docs` carries the endpoint with both examples.
`PostgresTestSuiteRunRepositoryFunctionalTests` gains a case that `findAllWithSnapshotRefs` returns the
refs while `findAll` still returns `suiteSnapshot == null`.

`MetaTestDataHelper.createTestSuiteRun` writes a snapshot **without** `deploymentRef`; add an
intent-named helper overload that seeds the refs. No raw SQL or snapshot JSON inside test methods.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.