jose-compu / jose-compu/funes.cpp

[Tests] Add CI integration for LongMemEval benchmark

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
C++
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary
Integrate the LongMemEval benchmark into CI/CD for regression detection.

## Current State
- `benchmark.py` - Rule-based proxy benchmark
- `llama_cli_ab_test.py` - Real LLM A/B testing
- `semantic_memory_accuracy_smoke.py` - Accuracy-focused tests
- Not currently running in CI

## Implementation Plan
1. **Create GitHub Actions workflow** `.github/workflows/longmemeval.yml`
2. **Select lightweight model** for CI (e.g., LFM2.5-1.2B Q4_K_M)
3. **Set success thresholds**:
- Memory ON accuracy > Memory OFF accuracy
- No regressions in any category
4. **Add caching** for downloaded models

## Workflow Design
```yaml
Trigger: [pull_request, workflow_dispatch]
Steps:
1. Build llama-cli with semantic memory
2. Download/cache test model
3. Run semantic_memory_smoke.py
4. Run semantic_memory_accuracy_smoke.py --fail-on-no-lift
5. Post results as PR comment
```

## Acceptance Criteria
- [ ] Workflow runs on every PR
- [ ] Results posted as PR comment
- [ ] Failed tests block merge (optional)
- [ ] Documentation in `tests/README.md`

## Related Files
- `tools/longmemeval-mini/benchmark.py`
- `tools/longmemeval-mini/llama_cli_ab_test.py`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.