epam / epam/ai-dial-admin-evaluation-framework-backend

[Eval] Add scheduled evaluation runs with cron-based recurrence and per-schedule DIAL API key

Open
#32 0 comments 0 reactions 1 assignee Claimed by @buhaiovos View on GitHub
enhancement
Dominant language
Java
Stars
3
Forks
1
Avg merge
1d 1h
Merged PRs (30d)
27

Description

### Name and Version

0.2.0

### What is the problem this feature will solve?

Today, every evaluation run must be triggered manually (via UI or API) by an authenticated user. This creates several gaps:

1. **No regression monitoring** — teams that want to catch quality regressions in a deployment over time must remember to launch the same eval suite repeatedly by hand, or build and operate their own external scheduler that calls EF's API.
2. **No unattended runs** — suites that should execute overnight, on weekends, or after model/prompt deploys cannot be set up to run without a human present.
3. **Identity coupling** — because runs are tied to a user JWT, there is no first-class way to attribute a recurring run to a stable, long-lived credential. When the original user leaves or rotates credentials, any scheduled-style automation built on top of EF breaks.

### What is the feature you are proposing to solve the problem?

Introduce **scheduled evaluation runs** as a first-class concept in EF. A user can attach a recurrence specification to an existing test suite, and EF will automatically create and execute runs on that schedule under a configured service-account-style identity, without further user interaction.

High-level behavior:

- **Schedule definition** — a schedule is a persistent entity that references an existing test suite and carries: a cron expression, a time zone, a per-schedule DIAL API key, an enabled/disabled flag, and metadata (owner, name, description).
- **Recurrence** — schedules use **cron expressions** for fire times. Time zone is configurable per schedule so calendar-based recurrences (e.g. "every weekday 02:00 in Europe/Warsaw") behave predictably across DST transitions.
- **Identity** — each schedule stores its own **DIAL API key**, supplied at schedule creation. Runs dispatched by the schedule execute under that key rather than a personal JWT, so the schedule's lifetime is decoupled from any individual user's session and credentials can be rotated centrally without touching the schedule definition.
- **Execution** — at each fire time, EF creates a new run for the referenced suite and dispatches it through the existing evaluation pipeline using the schedule's API key. Runs originating from a schedule are tagged with the schedule ID so they can be listed, filtered, and audited.
- **Lifecycle controls** — schedules can be paused, resumed, edited (cron, time zone, API key, target suite), and deleted. Disabling or deleting a schedule must not affect runs that have already been dispatched.
- **Concurrency policy** — when a previous scheduled run for the same schedule is still executing at the next fire time, the new run is either skipped or queued (configurable per schedule, with a safe default).
- **Observability** — schedule history (last fire time, next fire time, last run status, skip reason if any) is queryable via REST and visible in the UI.
- **Credential handling** — DIAL API keys are stored encrypted at rest, never returned in plaintext through the API, and validated at schedule creation/update time so users get immediate feedback on bad credentials.

> **Note — preferred direction to investigate first.** Before committing to an EF-specific scheduler, it is worth investigating whether a **general scheduler DIAL component** (a shared scheduling primitive owned by DIAL Core or a sibling DIAL service) can be introduced, with EF being just one of its consumers / use cases. A shared component would benefit other DIAL-ecosystem services that need recurring triggers, avoid duplicating cron parsing, persistence, dispatch, observability, and credential handling per service, and centralize operational concerns (HA, leader election, missed-fire policy, time-zone correctness). The EF-specific design described above should be treated as the **fallback** if a shared scheduler is not feasible in the relevant timeframe.

**Out of initial scope (deferred to a follow-up feature):**

- **Notifications of results** — emitting alerts on completion, regression, or failure (email, webhook, Slack/Teams, DIAL inbox) is intentionally not part of this feature. The scheduling primitive should land first; notification wiring can build on it later, since it has its own design space (channels, templating, thresholds, per-user routing).
- **Non-cron recurrence forms** — fixed intervals, calendar-style "every N business days", etc., are out of scope. Cron is expressive enough for the initial use cases.

### What alternatives have you considered?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.