Lightning-AI / Lightning-AI/litAI

Optional deterministic CI regression check (record/replay)?

Open
#73 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
52
Forks
12
PR merge metrics
No merged PRs in 30d

Description

Hi! I maintain RunLedger (https://github.com/runledger/Runledger), a small CLI for deterministic CI regression checks for tool-using agents (record once, replay in CI).

Would you be open to a small, optional PR that adds:

- `evals/runledger/` (suite + one case + schema + cassette)
- `baselines/runledger-demo.json`
- an optional GitHub Actions workflow to run the CI check (manual or on PR, depending on what you prefer)

It's small and removable, and I'm happy to close it if you don't want it.

The goal is to catch agent/tool regressions in CI without live tool calls (record once, replay in CI; fail on mismatch).

If you're interested, what is the best existing agent/example entrypoint in this repo to wire the suite to?

Contributor guide

No contributing guide indexed for this repository

Research direction

No existing entry point, test file, or CI configuration is named. Start by locating the repository's agent/example entrypoint and current GitHub Actions workflows, then determine where an optional replay suite could connect. Done would include the proposed evals/runledger/ files, baselines/runledger-demo.json, and an optional workflow that fails on replay mismatch without live tool calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.