Lightning-AI / Lightning-AI/litAI
Optional deterministic CI regression check (record/replay)?
- Dominant language
- Python
- Stars
- 52
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
Hi! I maintain RunLedger (https://github.com/runledger/Runledger), a small CLI for deterministic CI regression checks for tool-using agents (record once, replay in CI).
Would you be open to a small, optional PR that adds:
- `evals/runledger/` (suite + one case + schema + cassette)
- `baselines/runledger-demo.json`
- an optional GitHub Actions workflow to run the CI check (manual or on PR, depending on what you prefer)
It's small and removable, and I'm happy to close it if you don't want it.
The goal is to catch agent/tool regressions in CI without live tool calls (record once, replay in CI; fail on mismatch).
If you're interested, what is the best existing agent/example entrypoint in this repo to wire the suite to?
Contributor guide
No contributing guide indexed for this repository
Research direction
No existing entry point, test file, or CI configuration is named. Start by locating the repository's agent/example entrypoint and current GitHub Actions workflows, then determine where an optional replay suite could connect. Done would include the proposed evals/runledger/ files, baselines/runledger-demo.json, and an optional workflow that fails on replay mismatch without live tool calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100