epam / epam/ai-dial-admin-evaluation-framework-backend
[Eval] Query batch of test suite runs costs
- Dominant language
- Java
- Stars
- 3
- Forks
- 1
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 30
Description
# Page-scoped total run cost from dial-adas (one query for many runs)
## Context
`GET /api/v1/test-suite-runs/{id}/costs` today issues **two sequential dial-adas calls per run** (one per
`eval.phase`) and returns two per-call *averages* (`avgTestCaseCost`, `avgMetricEvalCost`). The upcoming
enriched run listing (see the follow-up issue) needs the **total** cost of many runs at once, and it must
fetch prices **exactly once per page — never once per run**.
Run cost lives only in dial-adas. Usage-log rows correlate to a run through a single flat baggage string:
```
eval.phase=execution,eval.run.id=,eval.suite.id=,run.index=0,testcase.id=
```
so a per-run breakdown requires dial-adas to derive the run id **inside** the query. Whether it can is
unverified: the archived design (`openspec/changes/archive/2026-08-20-test-suite-run-costs/design.md`,
Risks) explicitly flagged grouping over a `json_extract_string` expression as unconfirmed, and only the
`and` / `co` / `count` / `avg` aggregate shape has been validated against a real deployment.
Because `testcase.id` is part of the baggage, grouping by the raw baggage string is not an option — its
cardinality is one value per test case per phase.
## Scope
### Phase 1 — spike (blocking, do first)
Probe a real dial-adas with both shapes and record which one it accepts, the exact function names it
exposes, and whether `group_by` resolves a select alias.
**Shape A — group by an extracted run id** (preferred; query size independent of page size):
```json
{ "entity": "dial_usage_log", "mode": "aggregate",
"filter": {"op":"or","args":[
{"op":"co","args":[ {"type":"fn","name":"json_extract_string",
"args":[{"type":"field","name":"request_tags"},{"type":"value","value":"baggage"}]},
{"type":"value","value":"eval.run.id="} ]}
/* …one per run id on the page… */ ]},
"select": [
{"expr": {"type":"fn","name":"regexp_extract","args":[
/* the json_extract_string expression above */,
{"type":"value","value":"eval\\.run\\.id=([0-9a-f-]{36})"} ]}, "as": "run_id"},
{"expr": {"type":"fn","name":"sum","args":[{"type":"field","name":"total_price"}]}, "as": "total_cost"} ],
"group_by": ["run_id"] }
```
**Shape B — one conditional sum per run**, `group_by: []` (no grouping support needed, but the query
grows with page size):
```
select: [ sum_if(total_price, co(baggage, 'eval.run.id=')) as c_, … ]
```
**Outcome gate:** if neither shape works, close this issue with the finding documented and the enriched
listing ships **without** a cost field. Do **not** fall back to one call per run.
### Phase 2 — implementation (only if the spike succeeds)
- `RunCostQueryBuilder.buildPageTotalCostQuery(Collection runIds)` — reuse the existing private
`jsonExtractBaggage` / `baggageContains` / `stringValue` helpers. **No `eval.phase` predicate**, so the
total spans both phases and any phase added later. `buildAggregateQuery(runId, phase)` stays untouched.
- `client/dialadas/dto/AdasAggregateRowDto` — add `@JsonProperty("run_id") String runId` and
`@JsonProperty("total_cost") Double totalCost`, both nullable. Unknown-property tolerance keeps the
existing avg path unaffected.
- New `service/domain/RunCostFetcher`:
- `Map fetchTotalCosts(Collection runIds)` — **one**
`dialAdasClient.executeAggregate(...)` call, rows keyed back to run ids.
- `record RunCost(Double value, RunCostStatus status)` with
`RunCostStatus { AVAILABLE, NO_DATA, UNAVAILABLE }` — a bare `null` cannot distinguish "no usage rows
recorded" from "adas did not answer".
- A run absent from the response → `NO_DATA`. `catch (DialAdasClientException e)` → log with the
exception as the **trailing** SLF4J argument (`LoggingConventionTest`) → all runs on the page
`UNAVAILABLE`. **Never rethrow** — a listing must not fail because adas is down.
- No transaction; must never be invoked with a meta or analytics transaction open.
- Config (defaults in `application.yml` only; properties class holds structure + validation):
`test-suite-run.enriched-list.cost.enabled` / `TEST_SUITE_RUN_ENRICHED_COST_ENABLED`, default `true`.
Add the row to `docs/configuration.md` with all six columns and amend §5.5, which currently says
dial-adas is queried only by `/costs`.
## Out of scope
- Any change to `GET /api/v1/test-suite-runs/{id}/costs` — it keeps its two averages and its 502/504
failure contract.
- Per-test-case or per-phase cost breakdowns.
- Caching of adas responses.
## Acceptance criteria
- [ ] Spike outcome recorded in the OpenSpec change's `design.md`: which shape adas accepts, the exact
function names, and whether `group_by` resolves a select alias.
- [ ] Costs for **N runs are fetched in exactly one HTTP call**, asserted by a test
(`verify(dialAdasClient, times(1)).executeAggregate(any())`).
- [ ] Aggregation stays **server-side** (no row-mode fetch summed in application code).
- [ ] A run with no usage rows yields `NO_DATA` with a null value — never `0`.
- [ ] An adas timeout or connection failure yields `UNAVAILABLE` for the page and no exception escapes
`RunCostFetcher`.
- [ ] Existing `RunCostQueryBuilderTest` avg assertions still pass unchanged.
Manual: point `DIAL_ADAS_URL` at a real deployment and run the two spike payloads with `curl` against
`POST {base}/v1/queries/execute` before writing any code.
Contributor guide
Assessment
This issue has not been assessed yet.