aipotheosis-labs / aipotheosis-labs/aci
Add three-tier difficulty prompts for user intent evaluation
- Vorherrschende Sprache
- Python
- Sterne
- 4.9k
- Forks
- 484
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
### Required Pre-requisites
- [x] I have read the [Documentation](https://www.aci.dev/docs)
- [x] I have searched the [Issue Tracker](https://github.com/aipotheosis-labs/aci/issues) and [Discussions](https://github.com/aipotheosis-labs/aci/discussions) that this hasn't been reported yet.
- [x] Consider asking in [Discussions](https://github.com/aipotheosis-labs/aci/discussions) first
### Motivation
Motivation: Create a robust evaluation framework with three distinct difficulty levels of synthetic data generation to comprehensively test the user intent search pipeline's performance across various real-world scenarios.
### Proposed Solution
## 🛠️ Proposed Solution
**Implement a three-tier synthetic data generation system with progressive difficulty levels:**
### Core Implementation:
1. **Three Prompt Templates** (`prompt_easy`, `prompt_medium`, `prompt_hard`)
- **Easy**: Direct app mentions + clear goals
- **Medium**: App mentions + business context, no explicit function terms
- **Hard**: Implicit/contextual language, optional app mentions
2. **Graduated Complexity Testing**
- Baseline performance (easy) → Real-world scenarios (medium) → Edge cases (hard)
- Systematic evaluation across user communication styles
3. **Enhanced Evaluation Pipeline**
- Custom dataset naming by difficulty level
- Comparative performance analysis
- Comprehensive metrics tracking per tier
### Outcome:
A systematic evaluation framework that provides **data-driven insights** into search pipeline performance across realistic user complexity scenarios, enabling targeted improvements and benchmarking.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.