jose-compu / jose-compu/funes.cpp
[Tests] Add CI integration for LongMemEval benchmark
- Dominant language
- C++
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Integrate the LongMemEval benchmark into CI/CD for regression detection.
## Current State
- `benchmark.py` - Rule-based proxy benchmark
- `llama_cli_ab_test.py` - Real LLM A/B testing
- `semantic_memory_accuracy_smoke.py` - Accuracy-focused tests
- Not currently running in CI
## Implementation Plan
1. **Create GitHub Actions workflow** `.github/workflows/longmemeval.yml`
2. **Select lightweight model** for CI (e.g., LFM2.5-1.2B Q4_K_M)
3. **Set success thresholds**:
- Memory ON accuracy > Memory OFF accuracy
- No regressions in any category
4. **Add caching** for downloaded models
## Workflow Design
```yaml
Trigger: [pull_request, workflow_dispatch]
Steps:
1. Build llama-cli with semantic memory
2. Download/cache test model
3. Run semantic_memory_smoke.py
4. Run semantic_memory_accuracy_smoke.py --fail-on-no-lift
5. Post results as PR comment
```
## Acceptance Criteria
- [ ] Workflow runs on every PR
- [ ] Results posted as PR comment
- [ ] Failed tests block merge (optional)
- [ ] Documentation in `tests/README.md`
## Related Files
- `tools/longmemeval-mini/benchmark.py`
- `tools/longmemeval-mini/llama_cli_ab_test.py`
Contributor guide
Assessment
This issue has not been assessed yet.