epam / epam/ai-dial-admin-evaluation-framework-backend

[Eval] [Tech] Add ClickHouse as a selectable analytics datasource backend

Open
#34 0 comments 0 reactions 1 assignee Assigned to @buhaiovos View on GitHub
enhancement
Dominant language
Java
Stars
3
Forks
1
Avg merge
1d 1h
Merged PRs (30d)
30

Description

### Name and Version

0.1.0

### What is the problem this feature will solve?

The Evaluation Framework stores all analytics data — per-test-case run results, eval summaries, and run metric snapshots — in PostgreSQL. As eval volumes grow (large suites, frequent re-runs, longer retention windows), PostgreSQL becomes a bottleneck for the analytics workload:

1. **Aggregation and scan performance** — dashboards and exports that aggregate metrics across many runs, group by metric/dimension, or scan wide JSONB columns get progressively slower as the result tables grow into the tens or hundreds of millions of rows.
2. **Mismatch with workload shape** — analytics writes are append-only and reads are predominantly range scans and aggregations over time-ordered data. A row-store OLTP engine is not the optimal fit; a columnar OLAP engine is.
3. **No path to scale out** — PostgreSQL on the analytics side has to be vertically scaled, which limits how far the platform can grow without architectural changes.

### What is the feature you are proposing to solve the problem?

Add **ClickHouse** as an additional analytics backend selectable via the existing `datasource.analytics.vendor` configuration switch. PostgreSQL remains fully supported; operators choose the backend that fits their deployment.

High-level behavior:

- The existing `datasource.analytics.vendor` property gains a new value (e.g. `CLICKHOUSE`) alongside `POSTGRES`. Choosing it activates a ClickHouse-backed implementation of the analytics data layer; choosing PostgreSQL keeps current behavior unchanged.
- All analytics-side persistence — test case run results, eval summaries, run metric snapshots, computation versioning — is implemented against ClickHouse with schema and indexing tuned for columnar OLAP (sort keys on time + run/computation identifiers, append-only inserts, batched writes).
- The service-layer API, DTOs, cursor-based pagination contract, and REST endpoints are unchanged. Callers cannot tell which backend is in use.
- Schema management for ClickHouse is handled by a migration mechanism analogous to the existing Flyway-based PostgreSQL flow, kept in a separate, vendor-scoped location so the two backends evolve independently.
- Meta datasource (suites, runs metadata, configuration) stays on PostgreSQL — this change is scoped strictly to the analytics side.
- A documented migration path exists for moving historical analytics data from PostgreSQL to ClickHouse, but live dual-write or runtime switching between backends is **not** in scope; the vendor is chosen at deployment time.

No changes to public REST API contracts. No changes to the meta datasource schema. Existing PostgreSQL deployments are unaffected unless the operator opts in.

### What alternatives have you considered?

- **Stay on PostgreSQL and optimize harder** — partitioning, more aggressive indexing, materialized views, archiving old runs. Rejected as a long-term answer: buys time but does not change the fundamental row-store vs. columnar mismatch for aggregation-heavy analytics, and operational complexity grows quickly.
- **Introduce a different OLAP engine** (e.g. DuckDB, Apache Pinot, Druid). Rejected: ClickHouse is the best fit for the append-only, time-ordered, aggregation-heavy workload at the scale targeted, has mature operational tooling, and aligns with backends already used elsewhere in the ecosystem.
- **External data warehouse with periodic ETL** from PostgreSQL. Rejected: introduces ingestion lag (analytics must be near-real-time for in-progress runs), adds a second system to operate, and does no relieve pressure on the primary PostgreSQL analytics tables that EF itself reads from.
- **Replace PostgreSQL on the analytics side outright.** Rejected: many existing deployments are well-served by PostgreSQL and have no need for the operational overhead of running ClickHouse; coexistence behind the existing vendor switch is the lower-risk path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.