aipotheosis-labs / aipotheosis-labs/aci
Add three-tier difficulty prompts for user intent evaluation
- Langage dominant
- Python
- Étoiles
- 4.9k
- Forks
- 484
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
### Required Pre-requisites
- [x] I have read the [Documentation](https://www.aci.dev/docs)
- [x] I have searched the [Issue Tracker](https://github.com/aipotheosis-labs/aci/issues) and [Discussions](https://github.com/aipotheosis-labs/aci/discussions) that this hasn't been reported yet.
- [x] Consider asking in [Discussions](https://github.com/aipotheosis-labs/aci/discussions) first
### Motivation
Motivation: Create a robust evaluation framework with three distinct difficulty levels of synthetic data generation to comprehensively test the user intent search pipeline's performance across various real-world scenarios.
### Proposed Solution
## 🛠️ Proposed Solution
**Implement a three-tier synthetic data generation system with progressive difficulty levels:**
### Core Implementation:
1. **Three Prompt Templates** (`prompt_easy`, `prompt_medium`, `prompt_hard`)
- **Easy**: Direct app mentions + clear goals
- **Medium**: App mentions + business context, no explicit function terms
- **Hard**: Implicit/contextual language, optional app mentions
2. **Graduated Complexity Testing**
- Baseline performance (easy) → Real-world scenarios (medium) → Edge cases (hard)
- Systematic evaluation across user communication styles
3. **Enhanced Evaluation Pipeline**
- Custom dataset naming by difficulty level
- Comparative performance analysis
- Comprehensive metrics tracking per tier
### Outcome:
A systematic evaluation framework that provides **data-driven insights** into search pipeline performance across realistic user complexity scenarios, enabling targeted improvements and benchmarking.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.